2024-09-22T02:56:53.046 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-22T02:56:53.076 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-22T02:56:53.162 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-09-21_21:56:02-crimson-rados-squid-distro-crimson-smithi/7914725 branch: squid description: crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/radosbench-high-concurrency} email: ceph-qa@ceph.com first_in_suite: false job_id: '7914725' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi name: teuthology-2024-09-21_21:56:02-crimson-rados-squid-distro-crimson-smithi no_nested_subset: false os_type: centos os_version: 9.stream overrides: admin_socket: branch: squid ceph: conf: client.0: debug ms: 1 debug objecter: 20 debug rados: 20 global: enable experimental unrecoverable data corrupting features: crimson ms client mode: crc ms cluster mode: crc ms mon client mode: crc ms mon cluster mode: crc ms mon service mode: crc ms service mode: crc osd_pool_default_min_size: 2 osd_pool_default_size: 3 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 osd pool default crimson: true paxos service trim min: 10 osd: bluestore block size: 96636764160 bluestore compression algorithm: snappy bluestore compression mode: aggressive bluestore fsck on mount: true bluestore rocksdb cf: false crimson alien thread cpu cores: 6-7 crimson osd obc lru size: 10 debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 err to stderr: true log flush on exit: true log to file: false log to stderr: true mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd blocked scrub grace period: 3600 osd debug reject backfill probability: 0.3 osd delete sleep: 1 osd failsafe full ratio: 0.95 osd max backfills: 3 osd max markdown count: 1000 osd objectstore: bluestore osd scrub max interval: 120 osd scrub min interval: 60 osd shutdown pgref assert: true osd snap trim sleep: 2 osd.0: crimson seastar cpu cores: 0-2 osd.1: crimson seastar cpu cores: 3-5 osd.2: crimson seastar cpu cores: 0-2 osd.3: crimson seastar cpu cores: 3-5 flavor: crimson fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(POOL_APP_NOT_ENABLED\) - but it is still running - objects unfound and apparently lost - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - timeout on replica - late reservation from - MON_DOWN - OSDMAP_FLAGS - OSD_DOWN - PG_DEGRADED - PG_AVAILABILITY - POOL_APP_NOT_ENABLED - mons down - mon down - out of quorum - noscrub - nodeep-scrub sha1: 4e45e6b490e6cbe082cdf155fea780438ec53fc7 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: crimson sha1: 4e45e6b490e6cbe082cdf155fea780438ec53fc7 selinux: allowlist: - scontext=system_u:system_r:getty_t:s0 thrashosds: min_in: 4 workunit: branch: squid env: CRIMSON_COMPAT: '1' sha1: 4e45e6b490e6cbe082cdf155fea780438ec53fc7 owner: scheduled_teuthology@teuthology priority: 100 repo: https://git.ceph.com/ceph.git roles: - - mon.a - osd.0 - osd.1 - client.0 - node-exporter.a - - mgr.x - osd.2 - osd.3 - client.1 - prometheus.a - node-exporter.b seed: 1158 sha1: 4e45e6b490e6cbe082cdf155fea780438ec53fc7 sleep_before_teardown: 0 subset: 0/1 suite: crimson-rados suite_branch: squid suite_path: /home/teuthworker/src/git.ceph.com_ceph_4e45e6b490e6cbe082cdf155fea780438ec53fc7/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: 4e45e6b490e6cbe082cdf155fea780438ec53fc7 targets: smithi153.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFCluRB7uKXFDgmGsvjWZlrq9Js2ZTPKS44fcUbN38nl5K1EMgSiTKj5W7+NM+v5ASunsoio/RzSnHwUHDaBfsk= smithi196.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBErPbLRqq9iv4m0fjlIdE4UAlETUbC2/LEcIBoob52iILGgSQObGaIm5cb+1DtWPSF3r1ufr1Z2to8LCn7vuXgk= tasks: - install: null - ceph: conf: osd: debug monc: 20 flavor: crimson - thrashosds: ceph_objectstore_tool: false chance_inject_pause_short: 0 chance_reset_purged_snaps_last: 0 chance_thrash_cluster_full: 0 chance_thrash_pg_upmap: 0 dump_ops_enable: false min_in: 3 noscrub_toggle_delay: 0 reweight_osd: 0 sighup_delay: 0 thrash_primary_affinity: false timeout: 2400 - full_sequential: - radosbench: clients: - client.0 concurrency: 128 size: 8192 time: 90 - radosbench: clients: - client.0 concurrency: 128 size: 8192 time: 90 - radosbench: clients: - client.0 concurrency: 128 size: 8192 time: 90 - radosbench: clients: - client.0 concurrency: 128 size: 8192 time: 90 - radosbench: clients: - client.0 concurrency: 128 size: 8192 time: 90 - radosbench: clients: - client.0 concurrency: 128 size: 8192 time: 90 - radosbench: clients: - client.0 concurrency: 128 size: 8192 time: 90 - radosbench: clients: - client.0 concurrency: 128 size: 8192 time: 90 teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-21_21:56:02 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2902 2024-09-22T02:56:53.162 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_4e45e6b490e6cbe082cdf155fea780438ec53fc7/qa; will attempt to use it 2024-09-22T02:56:53.163 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_4e45e6b490e6cbe082cdf155fea780438ec53fc7/qa/tasks 2024-09-22T02:56:53.163 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-22T02:56:53.165 INFO:teuthology.task.internal:Checking packages... 2024-09-22T02:56:53.185 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'crimson' and ceph hash '4e45e6b490e6cbe082cdf155fea780438ec53fc7' 2024-09-22T02:56:53.185 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-22T02:56:53.185 INFO:teuthology.packaging:ref: None 2024-09-22T02:56:53.185 INFO:teuthology.packaging:tag: None 2024-09-22T02:56:53.185 INFO:teuthology.packaging:branch: squid 2024-09-22T02:56:53.185 INFO:teuthology.packaging:sha1: 4e45e6b490e6cbe082cdf155fea780438ec53fc7 2024-09-22T02:56:53.185 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&ref=squid 2024-09-22T02:56:53.380 INFO:teuthology.task.internal:Found packages for ceph version 19.1.1-373.g4e45e6b4 2024-09-22T02:56:53.382 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-22T02:56:53.388 INFO:teuthology.task.internal:no buildpackages task found 2024-09-22T02:56:53.388 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-22T02:56:53.437 INFO:teuthology.task.internal:Saving configuration 2024-09-22T02:56:53.454 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-22T02:56:53.493 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-22T02:56:53.519 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi153.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-09-21_21:56:02-crimson-rados-squid-distro-crimson-smithi/7914725', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-22 02:51:38.481892', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFCluRB7uKXFDgmGsvjWZlrq9Js2ZTPKS44fcUbN38nl5K1EMgSiTKj5W7+NM+v5ASunsoio/RzSnHwUHDaBfsk='} 2024-09-22T02:56:53.543 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi196.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-09-21_21:56:02-crimson-rados-squid-distro-crimson-smithi/7914725', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-22 02:51:38.480645', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBErPbLRqq9iv4m0fjlIdE4UAlETUbC2/LEcIBoob52iILGgSQObGaIm5cb+1DtWPSF3r1ufr1Z2to8LCn7vuXgk='} 2024-09-22T02:56:53.543 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-22T02:56:53.549 INFO:teuthology.task.internal:roles: ubuntu@smithi153.front.sepia.ceph.com - ['mon.a', 'osd.0', 'osd.1', 'client.0', 'node-exporter.a'] 2024-09-22T02:56:53.549 INFO:teuthology.task.internal:roles: ubuntu@smithi196.front.sepia.ceph.com - ['mgr.x', 'osd.2', 'osd.3', 'client.1', 'prometheus.a', 'node-exporter.b'] 2024-09-22T02:56:53.549 INFO:teuthology.run_tasks:Running task console_log... 2024-09-22T02:56:53.604 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f9b8f35c160>, signals=[15]) 2024-09-22T02:56:53.604 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-22T02:56:53.610 INFO:teuthology.task.internal:Opening connections... 2024-09-22T02:56:53.610 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi153.front.sepia.ceph.com 2024-09-22T02:56:53.611 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi153.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-22T02:56:53.689 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi196.front.sepia.ceph.com 2024-09-22T02:56:53.690 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi196.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-22T02:56:53.763 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-22T02:56:53.775 DEBUG:teuthology.orchestra.run.smithi153:> uname -m 2024-09-22T02:56:53.793 INFO:teuthology.orchestra.run.smithi153.stdout:x86_64 2024-09-22T02:56:53.794 DEBUG:teuthology.orchestra.run.smithi153:> cat /etc/os-release 2024-09-22T02:56:53.850 INFO:teuthology.orchestra.run.smithi153.stdout:NAME="CentOS Stream" 2024-09-22T02:56:53.851 INFO:teuthology.orchestra.run.smithi153.stdout:VERSION="9" 2024-09-22T02:56:53.851 INFO:teuthology.orchestra.run.smithi153.stdout:ID="centos" 2024-09-22T02:56:53.851 INFO:teuthology.orchestra.run.smithi153.stdout:ID_LIKE="rhel fedora" 2024-09-22T02:56:53.851 INFO:teuthology.orchestra.run.smithi153.stdout:VERSION_ID="9" 2024-09-22T02:56:53.851 INFO:teuthology.orchestra.run.smithi153.stdout:PLATFORM_ID="platform:el9" 2024-09-22T02:56:53.851 INFO:teuthology.orchestra.run.smithi153.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-22T02:56:53.851 INFO:teuthology.orchestra.run.smithi153.stdout:ANSI_COLOR="0;31" 2024-09-22T02:56:53.851 INFO:teuthology.orchestra.run.smithi153.stdout:LOGO="fedora-logo-icon" 2024-09-22T02:56:53.851 INFO:teuthology.orchestra.run.smithi153.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-22T02:56:53.851 INFO:teuthology.orchestra.run.smithi153.stdout:HOME_URL="https://centos.org/" 2024-09-22T02:56:53.851 INFO:teuthology.orchestra.run.smithi153.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-22T02:56:53.851 INFO:teuthology.orchestra.run.smithi153.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-22T02:56:53.851 INFO:teuthology.orchestra.run.smithi153.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-22T02:56:53.852 INFO:teuthology.lock.ops:Updating smithi153.front.sepia.ceph.com on lock server 2024-09-22T02:56:53.877 DEBUG:teuthology.orchestra.run.smithi196:> uname -m 2024-09-22T02:56:53.894 INFO:teuthology.orchestra.run.smithi196.stdout:x86_64 2024-09-22T02:56:53.894 DEBUG:teuthology.orchestra.run.smithi196:> cat /etc/os-release 2024-09-22T02:56:53.951 INFO:teuthology.orchestra.run.smithi196.stdout:NAME="CentOS Stream" 2024-09-22T02:56:53.951 INFO:teuthology.orchestra.run.smithi196.stdout:VERSION="9" 2024-09-22T02:56:53.951 INFO:teuthology.orchestra.run.smithi196.stdout:ID="centos" 2024-09-22T02:56:53.951 INFO:teuthology.orchestra.run.smithi196.stdout:ID_LIKE="rhel fedora" 2024-09-22T02:56:53.951 INFO:teuthology.orchestra.run.smithi196.stdout:VERSION_ID="9" 2024-09-22T02:56:53.951 INFO:teuthology.orchestra.run.smithi196.stdout:PLATFORM_ID="platform:el9" 2024-09-22T02:56:53.951 INFO:teuthology.orchestra.run.smithi196.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-22T02:56:53.952 INFO:teuthology.orchestra.run.smithi196.stdout:ANSI_COLOR="0;31" 2024-09-22T02:56:53.952 INFO:teuthology.orchestra.run.smithi196.stdout:LOGO="fedora-logo-icon" 2024-09-22T02:56:53.952 INFO:teuthology.orchestra.run.smithi196.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-22T02:56:53.952 INFO:teuthology.orchestra.run.smithi196.stdout:HOME_URL="https://centos.org/" 2024-09-22T02:56:53.952 INFO:teuthology.orchestra.run.smithi196.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-22T02:56:53.952 INFO:teuthology.orchestra.run.smithi196.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-22T02:56:53.952 INFO:teuthology.orchestra.run.smithi196.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-22T02:56:53.952 INFO:teuthology.lock.ops:Updating smithi196.front.sepia.ceph.com on lock server 2024-09-22T02:56:53.976 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-22T02:56:53.984 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-22T02:56:53.989 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-22T02:56:53.989 DEBUG:teuthology.orchestra.run.smithi153:> test '!' -e /home/ubuntu/cephtest 2024-09-22T02:56:53.991 DEBUG:teuthology.orchestra.run.smithi196:> test '!' -e /home/ubuntu/cephtest 2024-09-22T02:56:54.007 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-22T02:56:54.013 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-22T02:56:54.013 DEBUG:teuthology.orchestra.run.smithi153:> test -z $(ls -A /var/lib/ceph) 2024-09-22T02:56:54.050 DEBUG:teuthology.orchestra.run.smithi196:> test -z $(ls -A /var/lib/ceph) 2024-09-22T02:56:54.071 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-22T02:56:54.128 INFO:teuthology.run_tasks:Running task kernel... 2024-09-22T02:56:54.142 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-09-22T02:56:54.142 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}, 'mgr.x': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-09-22T02:56:54.142 DEBUG:teuthology.orchestra.run.smithi153:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-22T02:56:54.143 DEBUG:teuthology.orchestra.run.smithi196:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-22T02:56:54.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-22T02:56:54.159 DEBUG:teuthology.orchestra.run.smithi196:> uname -r 2024-09-22T02:56:54.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-22T02:56:54.162 DEBUG:teuthology.orchestra.run.smithi153:> uname -r 2024-09-22T02:56:54.215 INFO:teuthology.orchestra.run.smithi196.stdout:5.14.0-503.el9.x86_64 2024-09-22T02:56:54.215 INFO:teuthology.task.kernel:Running kernel on smithi196: 5.14.0-503.el9.x86_64 2024-09-22T02:56:54.215 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum install -y kernel 2024-09-22T02:56:54.219 INFO:teuthology.orchestra.run.smithi153.stdout:5.14.0-503.el9.x86_64 2024-09-22T02:56:54.219 INFO:teuthology.task.kernel:Running kernel on smithi153: 5.14.0-503.el9.x86_64 2024-09-22T02:56:54.219 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum install -y kernel 2024-09-22T02:56:56.766 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS Stream 9 - BaseOS 10 MB/s | 8.3 MB 00:00 2024-09-22T02:56:58.842 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS Stream 9 - AppStream 20 MB/s | 20 MB 00:01 2024-09-22T02:57:00.722 INFO:teuthology.orchestra.run.smithi196.stdout:CentOS Stream 9 - BaseOS 1.7 MB/s | 8.3 MB 00:04 2024-09-22T02:57:03.151 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS Stream 9 - CRB 12 MB/s | 6.5 MB 00:00 2024-09-22T02:57:04.802 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS Stream 9 - Extras packages 66 kB/s | 19 kB 00:00 2024-09-22T02:57:05.664 INFO:teuthology.orchestra.run.smithi153.stdout:Extra Packages for Enterprise Linux 31 MB/s | 23 MB 00:00 2024-09-22T02:57:09.631 INFO:teuthology.orchestra.run.smithi196.stdout:CentOS Stream 9 - AppStream 2.6 MB/s | 20 MB 00:07 2024-09-22T02:57:11.128 INFO:teuthology.orchestra.run.smithi153.stdout:lab-extras 33 kB/s | 1.7 kB 00:00 2024-09-22T02:57:12.968 INFO:teuthology.orchestra.run.smithi153.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-22T02:57:12.968 INFO:teuthology.orchestra.run.smithi153.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-22T02:57:13.044 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-09-22T02:57:13.054 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-09-22T02:57:13.054 INFO:teuthology.orchestra.run.smithi153.stdout: Package Architecture Version Repository Size 2024-09-22T02:57:13.054 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-09-22T02:57:13.054 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-09-22T02:57:13.054 INFO:teuthology.orchestra.run.smithi153.stdout: kernel x86_64 5.14.0-508.el9 baseos 36 k 2024-09-22T02:57:13.054 INFO:teuthology.orchestra.run.smithi153.stdout:Installing dependencies: 2024-09-22T02:57:13.054 INFO:teuthology.orchestra.run.smithi153.stdout: kernel-core x86_64 5.14.0-508.el9 baseos 16 M 2024-09-22T02:57:13.054 INFO:teuthology.orchestra.run.smithi153.stdout: kernel-modules x86_64 5.14.0-508.el9 baseos 35 M 2024-09-22T02:57:13.054 INFO:teuthology.orchestra.run.smithi153.stdout: kernel-modules-core x86_64 5.14.0-508.el9 baseos 29 M 2024-09-22T02:57:13.055 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-22T02:57:13.055 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-09-22T02:57:13.055 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-09-22T02:57:13.055 INFO:teuthology.orchestra.run.smithi153.stdout:Install 4 Packages 2024-09-22T02:57:13.055 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-22T02:57:13.056 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 79 M 2024-09-22T02:57:13.056 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 126 M 2024-09-22T02:57:13.056 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-09-22T02:57:13.421 INFO:teuthology.orchestra.run.smithi153.stdout:(1/4): kernel-5.14.0-508.el9.x86_64.rpm 323 kB/s | 36 kB 00:00 2024-09-22T02:57:14.023 INFO:teuthology.orchestra.run.smithi196.stdout:CentOS Stream 9 - CRB 10 MB/s | 6.5 MB 00:00 2024-09-22T02:57:15.481 INFO:teuthology.orchestra.run.smithi196.stdout:CentOS Stream 9 - Extras packages 78 kB/s | 19 kB 00:00 2024-09-22T02:57:16.519 INFO:teuthology.orchestra.run.smithi196.stdout:Extra Packages for Enterprise Linux 25 MB/s | 23 MB 00:00 2024-09-22T02:57:20.345 INFO:teuthology.orchestra.run.smithi153.stdout:(2/4): kernel-core-5.14.0-508.el9.x86_64.rpm 2.2 MB/s | 16 MB 00:07 2024-09-22T02:57:21.991 INFO:teuthology.orchestra.run.smithi196.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2024-09-22T02:57:23.889 INFO:teuthology.orchestra.run.smithi196.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-22T02:57:23.889 INFO:teuthology.orchestra.run.smithi196.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-22T02:57:23.977 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-09-22T02:57:23.983 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-09-22T02:57:23.983 INFO:teuthology.orchestra.run.smithi196.stdout: Package Architecture Version Repository Size 2024-09-22T02:57:23.983 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-09-22T02:57:23.983 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-09-22T02:57:23.983 INFO:teuthology.orchestra.run.smithi196.stdout: kernel x86_64 5.14.0-508.el9 baseos 36 k 2024-09-22T02:57:23.983 INFO:teuthology.orchestra.run.smithi196.stdout:Installing dependencies: 2024-09-22T02:57:23.983 INFO:teuthology.orchestra.run.smithi196.stdout: kernel-core x86_64 5.14.0-508.el9 baseos 16 M 2024-09-22T02:57:23.983 INFO:teuthology.orchestra.run.smithi196.stdout: kernel-modules x86_64 5.14.0-508.el9 baseos 35 M 2024-09-22T02:57:23.984 INFO:teuthology.orchestra.run.smithi196.stdout: kernel-modules-core x86_64 5.14.0-508.el9 baseos 29 M 2024-09-22T02:57:23.984 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-22T02:57:23.984 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-09-22T02:57:23.984 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-09-22T02:57:23.984 INFO:teuthology.orchestra.run.smithi196.stdout:Install 4 Packages 2024-09-22T02:57:23.984 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-22T02:57:23.984 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 79 M 2024-09-22T02:57:23.984 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 126 M 2024-09-22T02:57:23.985 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-09-22T02:57:24.376 INFO:teuthology.orchestra.run.smithi196.stdout:(1/4): kernel-5.14.0-508.el9.x86_64.rpm 123 kB/s | 36 kB 00:00 2024-09-22T02:57:26.292 INFO:teuthology.orchestra.run.smithi196.stdout:(2/4): kernel-modules-5.14.0-508.el9.x86_64.rpm 16 MB/s | 35 MB 00:02 2024-09-22T02:57:26.978 INFO:teuthology.orchestra.run.smithi153.stdout:(3/4): kernel-modules-core-5.14.0-508.el9.x86_6 2.1 MB/s | 29 MB 00:13 2024-09-22T02:57:27.109 INFO:teuthology.orchestra.run.smithi196.stdout:(3/4): kernel-modules-core-5.14.0-508.el9.x86_6 10 MB/s | 29 MB 00:02 2024-09-22T02:57:27.759 INFO:teuthology.orchestra.run.smithi196.stdout:(4/4): kernel-core-5.14.0-508.el9.x86_64.rpm 4.3 MB/s | 16 MB 00:03 2024-09-22T02:57:27.759 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-09-22T02:57:27.759 INFO:teuthology.orchestra.run.smithi196.stdout:Total 21 MB/s | 79 MB 00:03 2024-09-22T02:57:28.439 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-09-22T02:57:28.776 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-09-22T02:57:28.776 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-09-22T02:57:29.061 INFO:teuthology.orchestra.run.smithi153.stdout:(4/4): kernel-modules-5.14.0-508.el9.x86_64.rpm 2.2 MB/s | 35 MB 00:15 2024-09-22T02:57:29.061 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-09-22T02:57:29.062 INFO:teuthology.orchestra.run.smithi153.stdout:Total 4.9 MB/s | 79 MB 00:16 2024-09-22T02:57:29.743 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-09-22T02:57:29.750 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-09-22T02:57:29.750 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-09-22T02:57:30.107 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-09-22T02:57:30.107 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-09-22T02:57:31.081 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-09-22T02:57:31.082 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-09-22T02:57:32.241 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-09-22T02:57:33.003 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : kernel-modules-core-5.14.0-508.el9.x86_64 1/4 2024-09-22T02:57:33.086 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-22T02:57:33.533 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-09-22T02:57:34.232 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : kernel-modules-core-5.14.0-508.el9.x86_64 1/4 2024-09-22T02:57:34.313 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-22T02:57:35.846 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-22T02:57:35.979 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-22T02:57:37.103 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-22T02:57:37.241 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-22T02:57:40.833 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-22T02:57:40.879 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : kernel-5.14.0-508.el9.x86_64 4/4 2024-09-22T02:57:42.081 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-22T02:57:42.183 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : kernel-5.14.0-508.el9.x86_64 4/4 2024-09-22T02:57:45.766 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-22T02:57:47.051 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-22T02:58:23.711 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 4/4 2024-09-22T02:58:23.751 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 4/4 2024-09-22T02:58:24.056 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: kernel-5.14.0-508.el9.x86_64 4/4 2024-09-22T02:58:24.056 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : kernel-5.14.0-508.el9.x86_64 1/4 2024-09-22T02:58:24.056 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-22T02:58:24.056 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-22T02:58:24.895 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-22T02:58:24.895 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-22T02:58:24.895 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-09-22T02:58:24.895 INFO:teuthology.orchestra.run.smithi196.stdout: kernel-5.14.0-508.el9.x86_64 2024-09-22T02:58:24.895 INFO:teuthology.orchestra.run.smithi196.stdout: kernel-core-5.14.0-508.el9.x86_64 2024-09-22T02:58:24.895 INFO:teuthology.orchestra.run.smithi196.stdout: kernel-modules-5.14.0-508.el9.x86_64 2024-09-22T02:58:24.895 INFO:teuthology.orchestra.run.smithi196.stdout: kernel-modules-core-5.14.0-508.el9.x86_64 2024-09-22T02:58:24.895 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-22T02:58:24.896 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-09-22T02:58:25.156 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 4/4 2024-09-22T02:58:25.196 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 4/4 2024-09-22T02:58:25.271 DEBUG:teuthology.orchestra.run.smithi196:> rpm -q kernel | sort -rV | head -n 1 2024-09-22T02:58:25.364 INFO:teuthology.orchestra.run.smithi196.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-22T02:58:25.364 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-22T02:58:25.364 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-508.el9.x86_64 2024-09-22T02:58:25.364 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-22T02:58:25.364 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-22T02:58:25.364 INFO:teuthology.task.kernel:Installing distro kernel on mgr.x... 2024-09-22T02:58:25.365 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi196.front.sepia.ceph.com, path=None, version=distro) 2024-09-22T02:58:25.365 DEBUG:teuthology.orchestra.run.smithi196:> rpm -q kernel | sort -rV | head -n 1 2024-09-22T02:58:25.423 INFO:teuthology.orchestra.run.smithi196.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-22T02:58:25.424 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-22T02:58:25.424 DEBUG:teuthology.orchestra.run.smithi196:> sudo rpm -qi grub2-tools 2024-09-22T02:58:25.467 INFO:teuthology.orchestra.run.smithi196.stdout:Name : grub2-tools 2024-09-22T02:58:25.467 INFO:teuthology.orchestra.run.smithi196.stdout:Epoch : 1 2024-09-22T02:58:25.467 INFO:teuthology.orchestra.run.smithi196.stdout:Version : 2.06 2024-09-22T02:58:25.467 INFO:teuthology.orchestra.run.smithi196.stdout:Release : 82.el9 2024-09-22T02:58:25.467 INFO:teuthology.orchestra.run.smithi196.stdout:Architecture: x86_64 2024-09-22T02:58:25.467 INFO:teuthology.orchestra.run.smithi196.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-22T02:58:25.467 INFO:teuthology.orchestra.run.smithi196.stdout:Group : Unspecified 2024-09-22T02:58:25.467 INFO:teuthology.orchestra.run.smithi196.stdout:Size : 8274670 2024-09-22T02:58:25.467 INFO:teuthology.orchestra.run.smithi196.stdout:License : GPLv3+ 2024-09-22T02:58:25.467 INFO:teuthology.orchestra.run.smithi196.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-22T02:58:25.467 INFO:teuthology.orchestra.run.smithi196.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-22T02:58:25.467 INFO:teuthology.orchestra.run.smithi196.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-22T02:58:25.467 INFO:teuthology.orchestra.run.smithi196.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-22T02:58:25.468 INFO:teuthology.orchestra.run.smithi196.stdout:Packager : builder@centos.org 2024-09-22T02:58:25.468 INFO:teuthology.orchestra.run.smithi196.stdout:Vendor : CentOS 2024-09-22T02:58:25.468 INFO:teuthology.orchestra.run.smithi196.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-22T02:58:25.468 INFO:teuthology.orchestra.run.smithi196.stdout:Summary : Support tools for GRUB. 2024-09-22T02:58:25.468 INFO:teuthology.orchestra.run.smithi196.stdout:Description : 2024-09-22T02:58:25.468 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-22T02:58:25.468 INFO:teuthology.orchestra.run.smithi196.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-22T02:58:25.468 INFO:teuthology.orchestra.run.smithi196.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-22T02:58:25.468 INFO:teuthology.orchestra.run.smithi196.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-22T02:58:25.468 INFO:teuthology.orchestra.run.smithi196.stdout:hardware devices. 2024-09-22T02:58:25.468 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-22T02:58:25.468 INFO:teuthology.orchestra.run.smithi196.stdout:This subpackage provides tools for support of all platforms. 2024-09-22T02:58:25.469 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-22T02:58:25.469 INFO:teuthology.task.kernel:Updating grub on smithi196 to boot 5.14.0-508.el9.x86_64 2024-09-22T02:58:25.469 DEBUG:teuthology.orchestra.run.smithi196:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-22T02:58:25.509 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: kernel-5.14.0-508.el9.x86_64 4/4 2024-09-22T02:58:25.509 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : kernel-5.14.0-508.el9.x86_64 1/4 2024-09-22T02:58:25.510 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-22T02:58:25.510 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-22T02:58:26.130 INFO:teuthology.orchestra.run.smithi196.stderr:Generating grub configuration file ... 2024-09-22T02:58:26.343 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-22T02:58:26.344 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-22T02:58:26.344 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-09-22T02:58:26.344 INFO:teuthology.orchestra.run.smithi153.stdout: kernel-5.14.0-508.el9.x86_64 2024-09-22T02:58:26.344 INFO:teuthology.orchestra.run.smithi153.stdout: kernel-core-5.14.0-508.el9.x86_64 2024-09-22T02:58:26.344 INFO:teuthology.orchestra.run.smithi153.stdout: kernel-modules-5.14.0-508.el9.x86_64 2024-09-22T02:58:26.344 INFO:teuthology.orchestra.run.smithi153.stdout: kernel-modules-core-5.14.0-508.el9.x86_64 2024-09-22T02:58:26.344 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-22T02:58:26.344 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-09-22T02:58:26.689 DEBUG:teuthology.orchestra.run.smithi153:> rpm -q kernel | sort -rV | head -n 1 2024-09-22T02:58:26.778 INFO:teuthology.orchestra.run.smithi153.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-22T02:58:26.778 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-22T02:58:26.778 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-508.el9.x86_64 2024-09-22T02:58:26.778 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-22T02:58:26.778 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-22T02:58:26.778 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-09-22T02:58:26.778 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi153.front.sepia.ceph.com, path=None, version=distro) 2024-09-22T02:58:26.778 DEBUG:teuthology.orchestra.run.smithi153:> rpm -q kernel | sort -rV | head -n 1 2024-09-22T02:58:26.837 INFO:teuthology.orchestra.run.smithi153.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-22T02:58:26.837 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-22T02:58:26.837 DEBUG:teuthology.orchestra.run.smithi153:> sudo rpm -qi grub2-tools 2024-09-22T02:58:26.868 INFO:teuthology.orchestra.run.smithi153.stdout:Name : grub2-tools 2024-09-22T02:58:26.869 INFO:teuthology.orchestra.run.smithi153.stdout:Epoch : 1 2024-09-22T02:58:26.869 INFO:teuthology.orchestra.run.smithi153.stdout:Version : 2.06 2024-09-22T02:58:26.869 INFO:teuthology.orchestra.run.smithi153.stdout:Release : 82.el9 2024-09-22T02:58:26.869 INFO:teuthology.orchestra.run.smithi153.stdout:Architecture: x86_64 2024-09-22T02:58:26.869 INFO:teuthology.orchestra.run.smithi153.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-22T02:58:26.869 INFO:teuthology.orchestra.run.smithi153.stdout:Group : Unspecified 2024-09-22T02:58:26.869 INFO:teuthology.orchestra.run.smithi153.stdout:Size : 8274670 2024-09-22T02:58:26.869 INFO:teuthology.orchestra.run.smithi153.stdout:License : GPLv3+ 2024-09-22T02:58:26.869 INFO:teuthology.orchestra.run.smithi153.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-22T02:58:26.869 INFO:teuthology.orchestra.run.smithi153.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-22T02:58:26.869 INFO:teuthology.orchestra.run.smithi153.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-22T02:58:26.869 INFO:teuthology.orchestra.run.smithi153.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-22T02:58:26.869 INFO:teuthology.orchestra.run.smithi153.stdout:Packager : builder@centos.org 2024-09-22T02:58:26.869 INFO:teuthology.orchestra.run.smithi153.stdout:Vendor : CentOS 2024-09-22T02:58:26.869 INFO:teuthology.orchestra.run.smithi153.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-22T02:58:26.870 INFO:teuthology.orchestra.run.smithi153.stdout:Summary : Support tools for GRUB. 2024-09-22T02:58:26.870 INFO:teuthology.orchestra.run.smithi153.stdout:Description : 2024-09-22T02:58:26.870 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-22T02:58:26.870 INFO:teuthology.orchestra.run.smithi153.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-22T02:58:26.870 INFO:teuthology.orchestra.run.smithi153.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-22T02:58:26.870 INFO:teuthology.orchestra.run.smithi153.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-22T02:58:26.870 INFO:teuthology.orchestra.run.smithi153.stdout:hardware devices. 2024-09-22T02:58:26.870 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-22T02:58:26.870 INFO:teuthology.orchestra.run.smithi153.stdout:This subpackage provides tools for support of all platforms. 2024-09-22T02:58:26.871 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-22T02:58:26.871 INFO:teuthology.task.kernel:Updating grub on smithi153 to boot 5.14.0-508.el9.x86_64 2024-09-22T02:58:26.871 DEBUG:teuthology.orchestra.run.smithi153:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-22T02:58:27.475 INFO:teuthology.orchestra.run.smithi196.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-22T02:58:27.508 INFO:teuthology.orchestra.run.smithi196.stderr:done 2024-09-22T02:58:27.519 DEBUG:teuthology.orchestra.run.smithi196:> mktemp 2024-09-22T02:58:27.523 INFO:teuthology.orchestra.run.smithi153.stderr:Generating grub configuration file ... 2024-09-22T02:58:27.534 INFO:teuthology.orchestra.run.smithi196.stdout:/tmp/tmp.4rHkEH69Fk 2024-09-22T02:58:27.534 DEBUG:teuthology.orchestra.run.smithi196:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.4rHkEH69Fk 2024-09-22T02:58:27.606 DEBUG:teuthology.orchestra.run.smithi196:> sudo chmod 0666 /tmp/tmp.4rHkEH69Fk 2024-09-22T02:58:27.760 DEBUG:teuthology.orchestra.remote:smithi196:/tmp/tmp.4rHkEH69Fk is 6KB 2024-09-22T02:58:27.780 DEBUG:teuthology.orchestra.run.smithi196:> rm -fr /tmp/tmp.4rHkEH69Fk 2024-09-22T02:58:27.794 DEBUG:teuthology.orchestra.run.smithi196:> sudo /bin/ls /boot/loader/entries || true 2024-09-22T02:58:27.856 INFO:teuthology.orchestra.run.smithi196.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-22T02:58:27.856 INFO:teuthology.orchestra.run.smithi196.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-22T02:58:27.856 INFO:teuthology.orchestra.run.smithi196.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-22T02:58:27.856 INFO:teuthology.orchestra.run.smithi196.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64.conf 2024-09-22T02:58:27.858 DEBUG:teuthology.orchestra.run.smithi196:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64 2024-09-22T02:58:28.019 DEBUG:teuthology.orchestra.run.smithi196:> sudo shutdown -r now 2024-09-22T02:58:28.824 INFO:teuthology.orchestra.run.smithi153.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-22T02:58:28.852 INFO:teuthology.orchestra.run.smithi153.stderr:done 2024-09-22T02:58:28.858 DEBUG:teuthology.orchestra.run.smithi153:> mktemp 2024-09-22T02:58:28.874 INFO:teuthology.orchestra.run.smithi153.stdout:/tmp/tmp.XKS9q4ssZv 2024-09-22T02:58:28.874 DEBUG:teuthology.orchestra.run.smithi153:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.XKS9q4ssZv 2024-09-22T02:58:28.941 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod 0666 /tmp/tmp.XKS9q4ssZv 2024-09-22T02:58:29.092 DEBUG:teuthology.orchestra.remote:smithi153:/tmp/tmp.XKS9q4ssZv is 6KB 2024-09-22T02:58:29.140 DEBUG:teuthology.orchestra.run.smithi153:> rm -fr /tmp/tmp.XKS9q4ssZv 2024-09-22T02:58:29.155 DEBUG:teuthology.orchestra.run.smithi153:> sudo /bin/ls /boot/loader/entries || true 2024-09-22T02:58:29.228 INFO:teuthology.orchestra.run.smithi153.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-22T02:58:29.228 INFO:teuthology.orchestra.run.smithi153.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-22T02:58:29.228 INFO:teuthology.orchestra.run.smithi153.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-22T02:58:29.228 INFO:teuthology.orchestra.run.smithi153.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64.conf 2024-09-22T02:58:29.230 DEBUG:teuthology.orchestra.run.smithi153:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64 2024-09-22T02:58:29.391 DEBUG:teuthology.orchestra.run.smithi153:> sudo shutdown -r now 2024-09-22T02:58:58.047 INFO:teuthology.task.kernel:Checking client mgr.x for new kernel version... 2024-09-22T02:58:58.048 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi196.front.sepia.ceph.com' 2024-09-22T02:58:58.048 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi196.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-22T02:58:59.394 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-09-22T02:58:59.395 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi153.front.sepia.ceph.com' 2024-09-22T02:58:59.395 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi153.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-22T02:59:33.999 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.153 2024-09-22T02:59:43.006 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi153.front.sepia.ceph.com' 2024-09-22T02:59:43.007 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi153.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-22T02:59:58.065 DEBUG:teuthology.orchestra.remote:timed out 2024-09-22T03:00:01.391 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.153 2024-09-22T03:00:07.070 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi196.front.sepia.ceph.com' 2024-09-22T03:00:07.071 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi196.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-22T03:00:07.427 DEBUG:teuthology.orchestra.run.smithi196:> true 2024-09-22T03:00:07.828 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi196.front.sepia.ceph.com' 2024-09-22T03:00:07.829 INFO:teuthology.task.kernel:Checking kernel version of mgr.x, want "5.14.0-508.el9.x86_64"... 2024-09-22T03:00:07.829 DEBUG:teuthology.orchestra.run.smithi196:> uname -r 2024-09-22T03:00:07.843 INFO:teuthology.orchestra.run.smithi196.stdout:5.14.0-508.el9.x86_64 2024-09-22T03:00:07.843 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-508.el9.x86_64 vs 5.14.0-508.el9.x86_64 2024-09-22T03:00:07.843 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-22T03:00:07.843 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-22T03:00:08.844 INFO:teuthology.task.kernel:Enabling kdb on mgr.x... 2024-09-22T03:00:08.844 DEBUG:teuthology.orchestra.run.smithi196:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-22T03:00:08.936 INFO:teuthology.orchestra.run.smithi196.stdout:ttyS1 2024-09-22T03:00:08.974 DEBUG:teuthology.parallel:result is None 2024-09-22T03:00:13.391 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi153.front.sepia.ceph.com' 2024-09-22T03:00:13.655 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi153.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-22T03:00:16.945 DEBUG:teuthology.orchestra.run.smithi153:> true 2024-09-22T03:00:17.357 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi153.front.sepia.ceph.com' 2024-09-22T03:00:17.357 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-508.el9.x86_64"... 2024-09-22T03:00:17.357 DEBUG:teuthology.orchestra.run.smithi153:> uname -r 2024-09-22T03:00:17.375 INFO:teuthology.orchestra.run.smithi153.stdout:5.14.0-508.el9.x86_64 2024-09-22T03:00:17.375 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-508.el9.x86_64 vs 5.14.0-508.el9.x86_64 2024-09-22T03:00:17.375 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-22T03:00:17.375 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-22T03:00:18.377 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-09-22T03:00:18.379 DEBUG:teuthology.orchestra.run.smithi153:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-22T03:00:18.474 INFO:teuthology.orchestra.run.smithi153.stdout:ttyS1 2024-09-22T03:00:18.500 DEBUG:teuthology.parallel:result is None 2024-09-22T03:00:18.500 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-22T03:00:18.510 INFO:teuthology.task.internal:Creating test directory... 2024-09-22T03:00:18.510 DEBUG:teuthology.orchestra.run.smithi153:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-22T03:00:18.512 DEBUG:teuthology.orchestra.run.smithi196:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-22T03:00:18.531 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-22T03:00:18.537 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-22T03:00:18.546 INFO:teuthology.task.internal:Creating archive directory... 2024-09-22T03:00:18.546 DEBUG:teuthology.orchestra.run.smithi153:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-22T03:00:18.567 DEBUG:teuthology.orchestra.run.smithi196:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-22T03:00:18.859 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-22T03:00:18.866 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-22T03:00:18.867 DEBUG:teuthology.orchestra.run.smithi153:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-22T03:00:18.869 DEBUG:teuthology.orchestra.run.smithi196:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-22T03:00:18.900 INFO:teuthology.orchestra.run.smithi196.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-22T03:00:18.902 INFO:teuthology.orchestra.run.smithi153.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-22T03:00:18.912 INFO:teuthology.orchestra.run.smithi196.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-22T03:00:18.913 INFO:teuthology.orchestra.run.smithi153.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-22T03:00:18.914 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-22T03:00:18.925 INFO:teuthology.task.internal:Configuring sudo... 2024-09-22T03:00:18.925 DEBUG:teuthology.orchestra.run.smithi153:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-22T03:00:18.956 DEBUG:teuthology.orchestra.run.smithi196:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-22T03:00:18.984 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-22T03:00:18.993 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-22T03:00:18.993 DEBUG:teuthology.orchestra.run.smithi153:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-22T03:00:19.022 DEBUG:teuthology.orchestra.run.smithi196:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-22T03:00:19.041 DEBUG:teuthology.orchestra.run.smithi153:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-22T03:00:19.117 DEBUG:teuthology.orchestra.run.smithi153:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-22T03:00:19.159 DEBUG:teuthology.orchestra.run.smithi153:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-22T03:00:19.241 DEBUG:teuthology.orchestra.run.smithi153:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-22T03:00:19.282 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-09-22T03:00:19.282 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-22T03:00:19.350 DEBUG:teuthology.orchestra.run.smithi196:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-22T03:00:19.396 DEBUG:teuthology.orchestra.run.smithi196:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-22T03:00:19.445 DEBUG:teuthology.orchestra.run.smithi196:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-22T03:00:19.529 DEBUG:teuthology.orchestra.run.smithi196:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-22T03:00:19.573 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-09-22T03:00:19.573 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-22T03:00:19.641 DEBUG:teuthology.orchestra.run.smithi153:> sudo service rsyslog restart 2024-09-22T03:00:19.644 DEBUG:teuthology.orchestra.run.smithi196:> sudo service rsyslog restart 2024-09-22T03:00:19.724 INFO:teuthology.orchestra.run.smithi153.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-22T03:00:19.735 INFO:teuthology.orchestra.run.smithi196.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-22T03:00:20.045 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-22T03:00:20.056 INFO:teuthology.task.internal:Starting timer... 2024-09-22T03:00:20.057 INFO:teuthology.run_tasks:Running task pcp... 2024-09-22T03:00:20.171 INFO:teuthology.run_tasks:Running task selinux... 2024-09-22T03:00:20.203 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:getty_t:s0']} 2024-09-22T03:00:20.204 DEBUG:teuthology.orchestra.run.smithi153:> sudo service auditd rotate 2024-09-22T03:00:20.237 INFO:teuthology.orchestra.run.smithi153.stdout:Rotating logs: 2024-09-22T03:00:20.239 DEBUG:teuthology.orchestra.run.smithi196:> sudo service auditd rotate 2024-09-22T03:00:20.289 INFO:teuthology.orchestra.run.smithi196.stdout:Rotating logs: 2024-09-22T03:00:20.291 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-22T03:00:20.291 DEBUG:teuthology.orchestra.run.smithi153:> /usr/sbin/getenforce 2024-09-22T03:00:20.315 INFO:teuthology.orchestra.run.smithi153.stdout:Permissive 2024-09-22T03:00:20.315 DEBUG:teuthology.orchestra.run.smithi196:> /usr/sbin/getenforce 2024-09-22T03:00:20.338 INFO:teuthology.orchestra.run.smithi196.stdout:Permissive 2024-09-22T03:00:20.338 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi153.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi196.front.sepia.ceph.com': 'permissive'} 2024-09-22T03:00:20.338 DEBUG:teuthology.orchestra.run.smithi153:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-22T03:00:20.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-22T03:00:20.380 DEBUG:teuthology.orchestra.run.smithi196:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-22T03:00:20.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-22T03:00:20.408 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-22T03:00:20.408 DEBUG:teuthology.orchestra.run.smithi153:> sudo /usr/sbin/setenforce permissive 2024-09-22T03:00:20.451 DEBUG:teuthology.orchestra.run.smithi196:> sudo /usr/sbin/setenforce permissive 2024-09-22T03:00:20.486 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-22T03:00:20.496 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-09-22T03:00:20.498 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-09-22T03:00:20.655 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-22T03:00:20.666 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-22T03:00:20.667 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi153.front.sepia.ceph.com,smithi196.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-22T03:04:12.309 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi153.front.sepia.ceph.com'), Remote(name='ubuntu@smithi196.front.sepia.ceph.com')] 2024-09-22T03:04:12.310 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi153.front.sepia.ceph.com' 2024-09-22T03:04:12.310 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi153.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-22T03:04:12.386 DEBUG:teuthology.orchestra.run.smithi153:> true 2024-09-22T03:04:12.470 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi153.front.sepia.ceph.com' 2024-09-22T03:04:12.470 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi196.front.sepia.ceph.com' 2024-09-22T03:04:12.471 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi196.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-22T03:04:12.544 DEBUG:teuthology.orchestra.run.smithi196:> true 2024-09-22T03:04:12.617 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi196.front.sepia.ceph.com' 2024-09-22T03:04:12.618 INFO:teuthology.run_tasks:Running task clock... 2024-09-22T03:04:12.627 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-22T03:04:12.627 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-22T03:04:12.627 DEBUG:teuthology.orchestra.run.smithi153:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-22T03:04:12.630 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-22T03:04:12.630 DEBUG:teuthology.orchestra.run.smithi196:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-22T03:04:12.658 INFO:teuthology.orchestra.run.smithi153.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-22T03:04:12.672 INFO:teuthology.orchestra.run.smithi153.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-22T03:04:12.686 INFO:teuthology.orchestra.run.smithi196.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-22T03:04:12.697 INFO:teuthology.orchestra.run.smithi153.stderr:sudo: ntpd: command not found 2024-09-22T03:04:12.700 INFO:teuthology.orchestra.run.smithi196.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-22T03:04:12.708 INFO:teuthology.orchestra.run.smithi153.stdout:506 Cannot talk to daemon 2024-09-22T03:04:12.722 INFO:teuthology.orchestra.run.smithi153.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-22T03:04:12.725 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: ntpd: command not found 2024-09-22T03:04:12.736 INFO:teuthology.orchestra.run.smithi153.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-22T03:04:12.736 INFO:teuthology.orchestra.run.smithi196.stdout:506 Cannot talk to daemon 2024-09-22T03:04:12.751 INFO:teuthology.orchestra.run.smithi196.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-22T03:04:12.765 INFO:teuthology.orchestra.run.smithi196.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-22T03:04:12.801 INFO:teuthology.orchestra.run.smithi153.stderr:bash: line 1: ntpq: command not found 2024-09-22T03:04:12.806 INFO:teuthology.orchestra.run.smithi153.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-22T03:04:12.806 INFO:teuthology.orchestra.run.smithi153.stdout:=============================================================================== 2024-09-22T03:04:12.806 INFO:teuthology.orchestra.run.smithi153.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-22T03:04:12.806 INFO:teuthology.orchestra.run.smithi153.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-22T03:04:12.806 INFO:teuthology.orchestra.run.smithi153.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-22T03:04:12.806 INFO:teuthology.orchestra.run.smithi153.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-22T03:04:12.821 INFO:teuthology.orchestra.run.smithi196.stderr:bash: line 1: ntpq: command not found 2024-09-22T03:04:12.825 INFO:teuthology.orchestra.run.smithi196.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-22T03:04:12.825 INFO:teuthology.orchestra.run.smithi196.stdout:=============================================================================== 2024-09-22T03:04:12.825 INFO:teuthology.orchestra.run.smithi196.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-22T03:04:12.825 INFO:teuthology.orchestra.run.smithi196.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-22T03:04:12.825 INFO:teuthology.orchestra.run.smithi196.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-22T03:04:12.825 INFO:teuthology.orchestra.run.smithi196.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-22T03:04:12.826 INFO:teuthology.run_tasks:Running task install... 2024-09-22T03:04:12.834 DEBUG:teuthology.task.install:project ceph 2024-09-22T03:04:12.834 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'crimson', 'sha1': '4e45e6b490e6cbe082cdf155fea780438ec53fc7'}} 2024-09-22T03:04:12.834 DEBUG:teuthology.task.install:config {'flavor': 'crimson', 'sha1': '4e45e6b490e6cbe082cdf155fea780438ec53fc7'} 2024-09-22T03:04:12.834 INFO:teuthology.task.install:Using flavor: crimson 2024-09-22T03:04:12.839 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-22T03:04:12.839 INFO:teuthology.task.install:extra packages: [] 2024-09-22T03:04:12.840 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'crimson', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '4e45e6b490e6cbe082cdf155fea780438ec53fc7', 'tag': None, 'wait_for_package': False} 2024-09-22T03:04:12.840 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=4e45e6b490e6cbe082cdf155fea780438ec53fc7 2024-09-22T03:04:12.841 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'crimson', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '4e45e6b490e6cbe082cdf155fea780438ec53fc7', 'tag': None, 'wait_for_package': False} 2024-09-22T03:04:12.842 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=4e45e6b490e6cbe082cdf155fea780438ec53fc7 2024-09-22T03:04:12.988 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/centos/9/flavors/crimson/ 2024-09-22T03:04:12.988 INFO:teuthology.task.install.rpm:Package version is 19.1.1-373.g4e45e6b4 2024-09-22T03:04:12.992 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/centos/9/flavors/crimson/ 2024-09-22T03:04:12.992 INFO:teuthology.task.install.rpm:Package version is 19.1.1-373.g4e45e6b4 2024-09-22T03:04:13.210 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/centos/9/flavors/crimson/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/centos/9/flavors/crimson/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/centos/9/flavors/crimson/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-22T03:04:13.211 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-09-22T03:04:13.211 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-22T03:04:13.216 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/centos/9/flavors/crimson/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/centos/9/flavors/crimson/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/centos/9/flavors/crimson/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-22T03:04:13.216 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-09-22T03:04:13.216 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-22T03:04:13.237 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-09-22T03:04:13.237 DEBUG:teuthology.orchestra.run.smithi196:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/4e45e6b490e6cbe082cdf155fea780438ec53fc7/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-22T03:04:13.243 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-09-22T03:04:13.244 DEBUG:teuthology.orchestra.run.smithi153:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/4e45e6b490e6cbe082cdf155fea780438ec53fc7/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-22T03:04:13.304 DEBUG:teuthology.orchestra.run.smithi196:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-22T03:04:13.309 DEBUG:teuthology.orchestra.run.smithi153:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-22T03:04:13.380 DEBUG:teuthology.orchestra.run.smithi196:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-22T03:04:13.385 DEBUG:teuthology.orchestra.run.smithi153:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-22T03:04:13.443 INFO:teuthology.orchestra.run.smithi196.stdout:check_obsoletes = 1 2024-09-22T03:04:13.445 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum clean all 2024-09-22T03:04:13.449 INFO:teuthology.orchestra.run.smithi153.stdout:check_obsoletes = 1 2024-09-22T03:04:13.451 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum clean all 2024-09-22T03:04:13.672 INFO:teuthology.orchestra.run.smithi196.stdout:45 files removed 2024-09-22T03:04:13.680 INFO:teuthology.orchestra.run.smithi153.stdout:45 files removed 2024-09-22T03:04:13.700 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install ceph-radosgw 2024-09-22T03:04:13.706 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install ceph-radosgw 2024-09-22T03:04:14.430 INFO:teuthology.orchestra.run.smithi153.stdout:ceph packages for x86_64 198 kB/s | 87 kB 00:00 2024-09-22T03:04:14.430 INFO:teuthology.orchestra.run.smithi196.stdout:ceph packages for x86_64 196 kB/s | 87 kB 00:00 2024-09-22T03:04:15.054 INFO:teuthology.orchestra.run.smithi153.stdout:ceph noarch packages 20 kB/s | 12 kB 00:00 2024-09-22T03:04:15.055 INFO:teuthology.orchestra.run.smithi196.stdout:ceph noarch packages 20 kB/s | 12 kB 00:00 2024-09-22T03:04:15.780 INFO:teuthology.orchestra.run.smithi153.stdout:ceph source packages 2.9 kB/s | 2.0 kB 00:00 2024-09-22T03:04:15.798 INFO:teuthology.orchestra.run.smithi196.stdout:ceph source packages 2.8 kB/s | 2.0 kB 00:00 2024-09-22T03:04:16.313 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS Stream 9 - BaseOS 17 MB/s | 8.3 MB 00:00 2024-09-22T03:04:16.322 INFO:teuthology.orchestra.run.smithi196.stdout:CentOS Stream 9 - BaseOS 17 MB/s | 8.3 MB 00:00 2024-09-22T03:04:18.131 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS Stream 9 - AppStream 28 MB/s | 20 MB 00:00 2024-09-22T03:04:19.449 INFO:teuthology.orchestra.run.smithi196.stdout:CentOS Stream 9 - AppStream 9.9 MB/s | 20 MB 00:02 2024-09-22T03:04:23.849 INFO:teuthology.orchestra.run.smithi196.stdout:CentOS Stream 9 - CRB 11 MB/s | 6.5 MB 00:00 2024-09-22T03:04:24.131 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS Stream 9 - CRB 3.7 MB/s | 6.5 MB 00:01 2024-09-22T03:04:25.383 INFO:teuthology.orchestra.run.smithi196.stdout:CentOS Stream 9 - Extras packages 103 kB/s | 19 kB 00:00 2024-09-22T03:04:26.165 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS Stream 9 - Extras packages 27 kB/s | 19 kB 00:00 2024-09-22T03:04:26.429 INFO:teuthology.orchestra.run.smithi196.stdout:Extra Packages for Enterprise Linux 25 MB/s | 23 MB 00:00 2024-09-22T03:04:27.277 INFO:teuthology.orchestra.run.smithi153.stdout:Extra Packages for Enterprise Linux 23 MB/s | 23 MB 00:00 2024-09-22T03:04:31.818 INFO:teuthology.orchestra.run.smithi196.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2024-09-22T03:04:32.834 INFO:teuthology.orchestra.run.smithi153.stdout:lab-extras 32 kB/s | 1.7 kB 00:00 2024-09-22T03:04:34.044 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-09-22T03:04:34.052 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-09-22T03:04:34.052 INFO:teuthology.orchestra.run.smithi196.stdout: Package Arch Version Repository Size 2024-09-22T03:04:34.052 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-09-22T03:04:34.052 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-09-22T03:04:34.052 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-radosgw x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 9.9 M 2024-09-22T03:04:34.052 INFO:teuthology.orchestra.run.smithi196.stdout:Upgrading: 2024-09-22T03:04:34.052 INFO:teuthology.orchestra.run.smithi196.stdout: librados2 x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 3.2 M 2024-09-22T03:04:34.053 INFO:teuthology.orchestra.run.smithi196.stdout: librbd1 x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 3.1 M 2024-09-22T03:04:34.053 INFO:teuthology.orchestra.run.smithi196.stdout:Installing dependencies: 2024-09-22T03:04:34.053 INFO:teuthology.orchestra.run.smithi196.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-22T03:04:34.053 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-base x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 5.4 M 2024-09-22T03:04:34.053 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-common x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 19 M 2024-09-22T03:04:34.053 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-selinux x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 25 k 2024-09-22T03:04:34.053 INFO:teuthology.orchestra.run.smithi196.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-22T03:04:34.053 INFO:teuthology.orchestra.run.smithi196.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-22T03:04:34.053 INFO:teuthology.orchestra.run.smithi196.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-22T03:04:34.053 INFO:teuthology.orchestra.run.smithi196.stdout: libcephfs2 x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 760 k 2024-09-22T03:04:34.053 INFO:teuthology.orchestra.run.smithi196.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-22T03:04:34.053 INFO:teuthology.orchestra.run.smithi196.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-22T03:04:34.053 INFO:teuthology.orchestra.run.smithi196.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-22T03:04:34.054 INFO:teuthology.orchestra.run.smithi196.stdout: libradosstriper1 x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 501 k 2024-09-22T03:04:34.054 INFO:teuthology.orchestra.run.smithi196.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-22T03:04:34.054 INFO:teuthology.orchestra.run.smithi196.stdout: librgw2 x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 5.1 M 2024-09-22T03:04:34.054 INFO:teuthology.orchestra.run.smithi196.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-22T03:04:34.054 INFO:teuthology.orchestra.run.smithi196.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-22T03:04:34.054 INFO:teuthology.orchestra.run.smithi196.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-22T03:04:34.054 INFO:teuthology.orchestra.run.smithi196.stdout: python3-ceph-argparse x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 45 k 2024-09-22T03:04:34.054 INFO:teuthology.orchestra.run.smithi196.stdout: python3-ceph-common x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 136 k 2024-09-22T03:04:34.054 INFO:teuthology.orchestra.run.smithi196.stdout: python3-cephfs x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 161 k 2024-09-22T03:04:34.054 INFO:teuthology.orchestra.run.smithi196.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-22T03:04:34.054 INFO:teuthology.orchestra.run.smithi196.stdout: python3-rados x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 321 k 2024-09-22T03:04:34.054 INFO:teuthology.orchestra.run.smithi196.stdout: python3-rbd x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 301 k 2024-09-22T03:04:34.054 INFO:teuthology.orchestra.run.smithi196.stdout: python3-rgw x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 99 k 2024-09-22T03:04:34.055 INFO:teuthology.orchestra.run.smithi196.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-09-22T03:04:34.055 INFO:teuthology.orchestra.run.smithi196.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-09-22T03:04:34.055 INFO:teuthology.orchestra.run.smithi196.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-22T03:04:34.055 INFO:teuthology.orchestra.run.smithi196.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-22T03:04:34.055 INFO:teuthology.orchestra.run.smithi196.stdout:Installing weak dependencies: 2024-09-22T03:04:34.055 INFO:teuthology.orchestra.run.smithi196.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-09-22T03:04:34.055 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-22T03:04:34.055 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-09-22T03:04:34.055 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-09-22T03:04:34.055 INFO:teuthology.orchestra.run.smithi196.stdout:Install 30 Packages 2024-09-22T03:04:34.055 INFO:teuthology.orchestra.run.smithi196.stdout:Upgrade 2 Packages 2024-09-22T03:04:34.055 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-22T03:04:34.056 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 58 M 2024-09-22T03:04:34.056 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-09-22T03:04:35.043 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-09-22T03:04:35.073 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-09-22T03:04:35.073 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repository Size 2024-09-22T03:04:35.073 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-09-22T03:04:35.073 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-09-22T03:04:35.073 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-radosgw x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 9.9 M 2024-09-22T03:04:35.073 INFO:teuthology.orchestra.run.smithi153.stdout:Upgrading: 2024-09-22T03:04:35.073 INFO:teuthology.orchestra.run.smithi153.stdout: librados2 x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 3.2 M 2024-09-22T03:04:35.074 INFO:teuthology.orchestra.run.smithi153.stdout: librbd1 x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 3.1 M 2024-09-22T03:04:35.074 INFO:teuthology.orchestra.run.smithi153.stdout:Installing dependencies: 2024-09-22T03:04:35.074 INFO:teuthology.orchestra.run.smithi153.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-22T03:04:35.074 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-base x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 5.4 M 2024-09-22T03:04:35.074 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-common x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 19 M 2024-09-22T03:04:35.074 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-selinux x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 25 k 2024-09-22T03:04:35.074 INFO:teuthology.orchestra.run.smithi153.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-22T03:04:35.074 INFO:teuthology.orchestra.run.smithi153.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-22T03:04:35.074 INFO:teuthology.orchestra.run.smithi153.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-22T03:04:35.074 INFO:teuthology.orchestra.run.smithi153.stdout: libcephfs2 x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 760 k 2024-09-22T03:04:35.074 INFO:teuthology.orchestra.run.smithi153.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-22T03:04:35.074 INFO:teuthology.orchestra.run.smithi153.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-22T03:04:35.075 INFO:teuthology.orchestra.run.smithi153.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-22T03:04:35.075 INFO:teuthology.orchestra.run.smithi153.stdout: libradosstriper1 x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 501 k 2024-09-22T03:04:35.075 INFO:teuthology.orchestra.run.smithi153.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-22T03:04:35.075 INFO:teuthology.orchestra.run.smithi153.stdout: librgw2 x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 5.1 M 2024-09-22T03:04:35.075 INFO:teuthology.orchestra.run.smithi153.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-22T03:04:35.075 INFO:teuthology.orchestra.run.smithi153.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-22T03:04:35.075 INFO:teuthology.orchestra.run.smithi153.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-22T03:04:35.075 INFO:teuthology.orchestra.run.smithi153.stdout: python3-ceph-argparse x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 45 k 2024-09-22T03:04:35.075 INFO:teuthology.orchestra.run.smithi153.stdout: python3-ceph-common x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 136 k 2024-09-22T03:04:35.075 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cephfs x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 161 k 2024-09-22T03:04:35.075 INFO:teuthology.orchestra.run.smithi153.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-22T03:04:35.075 INFO:teuthology.orchestra.run.smithi153.stdout: python3-rados x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 321 k 2024-09-22T03:04:35.076 INFO:teuthology.orchestra.run.smithi153.stdout: python3-rbd x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 301 k 2024-09-22T03:04:35.076 INFO:teuthology.orchestra.run.smithi153.stdout: python3-rgw x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 99 k 2024-09-22T03:04:35.076 INFO:teuthology.orchestra.run.smithi153.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-09-22T03:04:35.076 INFO:teuthology.orchestra.run.smithi153.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-09-22T03:04:35.076 INFO:teuthology.orchestra.run.smithi153.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-22T03:04:35.076 INFO:teuthology.orchestra.run.smithi153.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-22T03:04:35.076 INFO:teuthology.orchestra.run.smithi153.stdout:Installing weak dependencies: 2024-09-22T03:04:35.076 INFO:teuthology.orchestra.run.smithi153.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-09-22T03:04:35.076 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-22T03:04:35.076 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-09-22T03:04:35.076 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-09-22T03:04:35.076 INFO:teuthology.orchestra.run.smithi153.stdout:Install 30 Packages 2024-09-22T03:04:35.076 INFO:teuthology.orchestra.run.smithi153.stdout:Upgrade 2 Packages 2024-09-22T03:04:35.076 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-22T03:04:35.077 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 58 M 2024-09-22T03:04:35.077 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-09-22T03:04:35.444 INFO:teuthology.orchestra.run.smithi196.stdout:(1/32): ceph-base-19.1.1-373.g4e45e6b4.el9.x86_ 4.1 MB/s | 5.4 MB 00:01 2024-09-22T03:04:35.536 INFO:teuthology.orchestra.run.smithi196.stdout:(2/32): ceph-selinux-19.1.1-373.g4e45e6b4.el9.x 275 kB/s | 25 kB 00:00 2024-09-22T03:04:35.761 INFO:teuthology.orchestra.run.smithi196.stdout:(3/32): libcephfs2-19.1.1-373.g4e45e6b4.el9.x86 3.3 MB/s | 760 kB 00:00 2024-09-22T03:04:35.953 INFO:teuthology.orchestra.run.smithi196.stdout:(4/32): libradosstriper1-19.1.1-373.g4e45e6b4.e 2.6 MB/s | 501 kB 00:00 2024-09-22T03:04:36.711 INFO:teuthology.orchestra.run.smithi196.stdout:(5/32): librgw2-19.1.1-373.g4e45e6b4.el9.x86_64 6.7 MB/s | 5.1 MB 00:00 2024-09-22T03:04:36.811 INFO:teuthology.orchestra.run.smithi196.stdout:(6/32): ceph-radosgw-19.1.1-373.g4e45e6b4.el9.x 3.7 MB/s | 9.9 MB 00:02 2024-09-22T03:04:36.836 INFO:teuthology.orchestra.run.smithi196.stdout:(7/32): python3-ceph-argparse-19.1.1-373.g4e45e 361 kB/s | 45 kB 00:00 2024-09-22T03:04:36.903 INFO:teuthology.orchestra.run.smithi196.stdout:(8/32): python3-ceph-common-19.1.1-373.g4e45e6b 1.4 MB/s | 136 kB 00:00 2024-09-22T03:04:36.928 INFO:teuthology.orchestra.run.smithi196.stdout:(9/32): python3-cephfs-19.1.1-373.g4e45e6b4.el9 1.7 MB/s | 161 kB 00:00 2024-09-22T03:04:37.065 INFO:teuthology.orchestra.run.smithi196.stdout:(10/32): python3-rados-19.1.1-373.g4e45e6b4.el9 1.9 MB/s | 321 kB 00:00 2024-09-22T03:04:37.104 INFO:teuthology.orchestra.run.smithi196.stdout:(11/32): python3-rbd-19.1.1-373.g4e45e6b4.el9.x 1.7 MB/s | 301 kB 00:00 2024-09-22T03:04:37.146 INFO:teuthology.orchestra.run.smithi196.stdout:(12/32): python3-rgw-19.1.1-373.g4e45e6b4.el9.x 1.2 MB/s | 99 kB 00:00 2024-09-22T03:04:37.372 INFO:teuthology.orchestra.run.smithi196.stdout:(13/32): ceph-common-19.1.1-373.g4e45e6b4.el9.x 6.0 MB/s | 19 MB 00:03 2024-09-22T03:04:37.397 INFO:teuthology.orchestra.run.smithi196.stdout:(14/32): boost-program-options-1.75.0-8.el9.x86 366 kB/s | 107 kB 00:00 2024-09-22T03:04:37.439 INFO:teuthology.orchestra.run.smithi196.stdout:(15/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 548 kB/s | 160 kB 00:00 2024-09-22T03:04:37.472 INFO:teuthology.orchestra.run.smithi196.stdout:(16/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 451 kB/s | 45 kB 00:00 2024-09-22T03:04:37.506 INFO:teuthology.orchestra.run.smithi196.stdout:(17/32): librdkafka-1.6.1-102.el9.x86_64.rpm 5.9 MB/s | 662 kB 00:00 2024-09-22T03:04:37.531 INFO:teuthology.orchestra.run.smithi196.stdout:(18/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.1 MB/s | 292 kB 00:00 2024-09-22T03:04:37.556 INFO:teuthology.orchestra.run.smithi196.stdout:(19/32): python3-prettytable-0.7.2-27.el9.noarc 504 kB/s | 42 kB 00:00 2024-09-22T03:04:37.582 INFO:teuthology.orchestra.run.smithi196.stdout:(20/32): qatlib-service-24.02.0-1.el9.x86_64.rp 715 kB/s | 36 kB 00:00 2024-09-22T03:04:37.607 INFO:teuthology.orchestra.run.smithi196.stdout:(21/32): qatlib-24.02.0-1.el9.x86_64.rpm 2.1 MB/s | 221 kB 00:00 2024-09-22T03:04:37.640 INFO:teuthology.orchestra.run.smithi196.stdout:(22/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 566 kB/s | 47 kB 00:00 2024-09-22T03:04:37.691 INFO:teuthology.orchestra.run.smithi196.stdout:(23/32): gperftools-libs-2.9.1-3.el9.x86_64.rpm 2.8 MB/s | 308 kB 00:00 2024-09-22T03:04:37.724 INFO:teuthology.orchestra.run.smithi196.stdout:(24/32): libarrow-doc-9.0.0-11.el9.noarch.rpm 352 kB/s | 26 kB 00:00 2024-09-22T03:04:37.750 INFO:teuthology.orchestra.run.smithi196.stdout:(25/32): liboath-2.6.7-2.el9.x86_64.rpm 854 kB/s | 49 kB 00:00 2024-09-22T03:04:37.783 INFO:teuthology.orchestra.run.smithi196.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 2.0 MB/s | 67 kB 00:00 2024-09-22T03:04:37.825 INFO:teuthology.orchestra.run.smithi196.stdout:(27/32): re2-20211101-20.el9.x86_64.rpm 4.5 MB/s | 191 kB 00:00 2024-09-22T03:04:37.867 INFO:teuthology.orchestra.run.smithi196.stdout:(28/32): parquet-libs-9.0.0-11.el9.x86_64.rpm 7.0 MB/s | 839 kB 00:00 2024-09-22T03:04:37.943 INFO:teuthology.orchestra.run.smithi196.stdout:(29/32): thrift-0.15.0-2.el9.x86_64.rpm 14 MB/s | 1.6 MB 00:00 2024-09-22T03:04:37.992 INFO:teuthology.orchestra.run.smithi153.stdout:(1/32): ceph-radosgw-19.1.1-373.g4e45e6b4.el9.x 3.8 MB/s | 9.9 MB 00:02 2024-09-22T03:04:38.043 INFO:teuthology.orchestra.run.smithi196.stdout:(30/32): libarrow-9.0.0-11.el9.x86_64.rpm 10 MB/s | 4.4 MB 00:00 2024-09-22T03:04:38.059 INFO:teuthology.orchestra.run.smithi153.stdout:(2/32): ceph-selinux-19.1.1-373.g4e45e6b4.el9.x 378 kB/s | 25 kB 00:00 2024-09-22T03:04:38.235 INFO:teuthology.orchestra.run.smithi153.stdout:(3/32): libcephfs2-19.1.1-373.g4e45e6b4.el9.x86 4.2 MB/s | 760 kB 00:00 2024-09-22T03:04:38.352 INFO:teuthology.orchestra.run.smithi153.stdout:(4/32): libradosstriper1-19.1.1-373.g4e45e6b4.e 4.2 MB/s | 501 kB 00:00 2024-09-22T03:04:38.376 INFO:teuthology.orchestra.run.smithi196.stdout:(31/32): librbd1-19.1.1-373.g4e45e6b4.el9.x86_6 7.2 MB/s | 3.1 MB 00:00 2024-09-22T03:04:38.427 INFO:teuthology.orchestra.run.smithi196.stdout:(32/32): librados2-19.1.1-373.g4e45e6b4.el9.x86 5.7 MB/s | 3.2 MB 00:00 2024-09-22T03:04:38.431 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-09-22T03:04:38.432 INFO:teuthology.orchestra.run.smithi196.stdout:Total 13 MB/s | 58 MB 00:04 2024-09-22T03:04:38.534 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-09-22T03:04:38.569 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-09-22T03:04:38.569 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-09-22T03:04:38.769 INFO:teuthology.orchestra.run.smithi153.stdout:(5/32): ceph-common-19.1.1-373.g4e45e6b4.el9.x8 5.7 MB/s | 19 MB 00:03 2024-09-22T03:04:38.835 INFO:teuthology.orchestra.run.smithi153.stdout:(6/32): ceph-base-19.1.1-373.g4e45e6b4.el9.x86_ 1.6 MB/s | 5.4 MB 00:03 2024-09-22T03:04:38.852 INFO:teuthology.orchestra.run.smithi153.stdout:(7/32): python3-ceph-argparse-19.1.1-373.g4e45e 542 kB/s | 45 kB 00:00 2024-09-22T03:04:38.927 INFO:teuthology.orchestra.run.smithi153.stdout:(8/32): librgw2-19.1.1-373.g4e45e6b4.el9.x86_64 8.8 MB/s | 5.1 MB 00:00 2024-09-22T03:04:38.952 INFO:teuthology.orchestra.run.smithi153.stdout:(9/32): python3-ceph-common-19.1.1-373.g4e45e6b 1.1 MB/s | 136 kB 00:00 2024-09-22T03:04:38.978 INFO:teuthology.orchestra.run.smithi153.stdout:(10/32): python3-cephfs-19.1.1-373.g4e45e6b4.el 1.3 MB/s | 161 kB 00:00 2024-09-22T03:04:39.003 INFO:teuthology.orchestra.run.smithi153.stdout:(11/32): python3-rados-19.1.1-373.g4e45e6b4.el9 4.2 MB/s | 321 kB 00:00 2024-09-22T03:04:39.028 INFO:teuthology.orchestra.run.smithi153.stdout:(12/32): python3-rgw-19.1.1-373.g4e45e6b4.el9.x 1.9 MB/s | 99 kB 00:00 2024-09-22T03:04:39.053 INFO:teuthology.orchestra.run.smithi153.stdout:(13/32): python3-rbd-19.1.1-373.g4e45e6b4.el9.x 2.9 MB/s | 301 kB 00:00 2024-09-22T03:04:39.196 INFO:teuthology.orchestra.run.smithi153.stdout:(14/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 319 kB/s | 45 kB 00:00 2024-09-22T03:04:39.221 INFO:teuthology.orchestra.run.smithi153.stdout:(15/32): boost-program-options-1.75.0-8.el9.x86 492 kB/s | 107 kB 00:00 2024-09-22T03:04:39.245 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-09-22T03:04:39.245 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-09-22T03:04:39.246 INFO:teuthology.orchestra.run.smithi153.stdout:(16/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 737 kB/s | 160 kB 00:00 2024-09-22T03:04:39.296 INFO:teuthology.orchestra.run.smithi153.stdout:(17/32): python3-prettytable-0.7.2-27.el9.noarc 846 kB/s | 42 kB 00:00 2024-09-22T03:04:39.330 INFO:teuthology.orchestra.run.smithi153.stdout:(18/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.6 MB/s | 292 kB 00:00 2024-09-22T03:04:39.388 INFO:teuthology.orchestra.run.smithi153.stdout:(19/32): librdkafka-1.6.1-102.el9.x86_64.rpm 3.4 MB/s | 662 kB 00:00 2024-09-22T03:04:39.414 INFO:teuthology.orchestra.run.smithi153.stdout:(20/32): qatlib-service-24.02.0-1.el9.x86_64.rp 431 kB/s | 36 kB 00:00 2024-09-22T03:04:39.439 INFO:teuthology.orchestra.run.smithi153.stdout:(21/32): qatlib-24.02.0-1.el9.x86_64.rpm 1.5 MB/s | 221 kB 00:00 2024-09-22T03:04:39.473 INFO:teuthology.orchestra.run.smithi153.stdout:(22/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 563 kB/s | 47 kB 00:00 2024-09-22T03:04:39.531 INFO:teuthology.orchestra.run.smithi153.stdout:(23/32): gperftools-libs-2.9.1-3.el9.x86_64.rpm 2.6 MB/s | 308 kB 00:00 2024-09-22T03:04:39.557 INFO:teuthology.orchestra.run.smithi153.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 2.1 MB/s | 49 kB 00:00 2024-09-22T03:04:39.582 INFO:teuthology.orchestra.run.smithi153.stdout:(25/32): libarrow-doc-9.0.0-11.el9.noarch.rpm 276 kB/s | 26 kB 00:00 2024-09-22T03:04:39.599 INFO:teuthology.orchestra.run.smithi153.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.6 MB/s | 67 kB 00:00 2024-09-22T03:04:39.632 INFO:teuthology.orchestra.run.smithi153.stdout:(27/32): re2-20211101-20.el9.x86_64.rpm 5.7 MB/s | 191 kB 00:00 2024-09-22T03:04:39.691 INFO:teuthology.orchestra.run.smithi153.stdout:(28/32): parquet-libs-9.0.0-11.el9.x86_64.rpm 7.5 MB/s | 839 kB 00:00 2024-09-22T03:04:39.733 INFO:teuthology.orchestra.run.smithi153.stdout:(29/32): thrift-0.15.0-2.el9.x86_64.rpm 16 MB/s | 1.6 MB 00:00 2024-09-22T03:04:39.825 INFO:teuthology.orchestra.run.smithi153.stdout:(30/32): libarrow-9.0.0-11.el9.x86_64.rpm 11 MB/s | 4.4 MB 00:00 2024-09-22T03:04:40.109 INFO:teuthology.orchestra.run.smithi153.stdout:(31/32): librbd1-19.1.1-373.g4e45e6b4.el9.x86_6 8.3 MB/s | 3.1 MB 00:00 2024-09-22T03:04:40.684 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-09-22T03:04:40.967 INFO:teuthology.orchestra.run.smithi153.stdout:(32/32): librados2-19.1.1-373.g4e45e6b4.el9.x86 2.5 MB/s | 3.2 MB 00:01 2024-09-22T03:04:40.971 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-09-22T03:04:40.971 INFO:teuthology.orchestra.run.smithi153.stdout:Total 9.8 MB/s | 58 MB 00:05 2024-09-22T03:04:41.072 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-09-22T03:04:41.107 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-09-22T03:04:41.108 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-09-22T03:04:41.109 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-09-22T03:04:41.170 INFO:teuthology.orchestra.run.smithi196.stdout: Upgrading : librados2-2:19.1.1-373.g4e45e6b4.el9.x86_64 2/34 2024-09-22T03:04:41.266 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: librados2-2:19.1.1-373.g4e45e6b4.el9.x86_64 2/34 2024-09-22T03:04:41.371 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-rados-2:19.1.1-373.g4e45e6b4.el9.x86_64 3/34 2024-09-22T03:04:41.396 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : libcephfs2-2:19.1.1-373.g4e45e6b4.el9.x86_64 4/34 2024-09-22T03:04:41.463 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: libcephfs2-2:19.1.1-373.g4e45e6b4.el9.x86_64 4/34 2024-09-22T03:04:41.492 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/34 2024-09-22T03:04:41.518 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 6/34 2024-09-22T03:04:41.540 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 7/34 2024-09-22T03:04:41.571 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 7/34 2024-09-22T03:04:41.771 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-09-22T03:04:41.772 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-09-22T03:04:41.800 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 8/34 2024-09-22T03:04:41.829 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 8/34 2024-09-22T03:04:41.858 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : liboath-2.6.7-2.el9.x86_64 9/34 2024-09-22T03:04:41.902 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-ceph-argparse-2:19.1.1-373.g4e45e6b4.el9.x 10/34 2024-09-22T03:04:41.923 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-cephfs-2:19.1.1-373.g4e45e6b4.el9.x86_64 11/34 2024-09-22T03:04:41.994 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 12/34 2024-09-22T03:04:42.020 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : libradosstriper1-2:19.1.1-373.g4e45e6b4.el9.x86_64 13/34 2024-09-22T03:04:42.120 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: libradosstriper1-2:19.1.1-373.g4e45e6b4.el9.x86_64 13/34 2024-09-22T03:04:42.311 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : thrift-0.15.0-2.el9.x86_64 14/34 2024-09-22T03:04:42.432 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : re2-1:20211101-20.el9.x86_64 15/34 2024-09-22T03:04:42.470 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 16/34 2024-09-22T03:04:42.511 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-09-22T03:04:42.543 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 18/34 2024-09-22T03:04:42.573 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 19/34 2024-09-22T03:04:42.606 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-09-22T03:04:42.947 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-09-22T03:04:42.985 INFO:teuthology.orchestra.run.smithi196.stdout: Upgrading : librbd1-2:19.1.1-373.g4e45e6b4.el9.x86_64 22/34 2024-09-22T03:04:43.059 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: librbd1-2:19.1.1-373.g4e45e6b4.el9.x86_64 22/34 2024-09-22T03:04:43.089 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-rbd-2:19.1.1-373.g4e45e6b4.el9.x86_64 23/34 2024-09-22T03:04:43.126 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-09-22T03:04:43.236 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-09-22T03:04:43.613 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 25/34 2024-09-22T03:04:43.637 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : librgw2-2:19.1.1-373.g4e45e6b4.el9.x86_64 26/34 2024-09-22T03:04:43.698 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-09-22T03:04:43.709 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: librgw2-2:19.1.1-373.g4e45e6b4.el9.x86_64 26/34 2024-09-22T03:04:43.777 INFO:teuthology.orchestra.run.smithi153.stdout: Upgrading : librados2-2:19.1.1-373.g4e45e6b4.el9.x86_64 2/34 2024-09-22T03:04:43.786 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-rgw-2:19.1.1-373.g4e45e6b4.el9.x86_64 27/34 2024-09-22T03:04:43.814 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-ceph-common-2:19.1.1-373.g4e45e6b4.el9.x86 28/34 2024-09-22T03:04:43.857 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: librados2-2:19.1.1-373.g4e45e6b4.el9.x86_64 2/34 2024-09-22T03:04:43.960 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-rados-2:19.1.1-373.g4e45e6b4.el9.x86_64 3/34 2024-09-22T03:04:43.995 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libcephfs2-2:19.1.1-373.g4e45e6b4.el9.x86_64 4/34 2024-09-22T03:04:44.077 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: libcephfs2-2:19.1.1-373.g4e45e6b4.el9.x86_64 4/34 2024-09-22T03:04:44.108 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/34 2024-09-22T03:04:44.141 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 6/34 2024-09-22T03:04:44.164 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 7/34 2024-09-22T03:04:44.191 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 7/34 2024-09-22T03:04:44.399 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 8/34 2024-09-22T03:04:44.436 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 8/34 2024-09-22T03:04:44.465 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : liboath-2.6.7-2.el9.x86_64 9/34 2024-09-22T03:04:44.509 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-ceph-argparse-2:19.1.1-373.g4e45e6b4.el9.x 10/34 2024-09-22T03:04:44.538 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-cephfs-2:19.1.1-373.g4e45e6b4.el9.x86_64 11/34 2024-09-22T03:04:44.615 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 12/34 2024-09-22T03:04:44.643 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libradosstriper1-2:19.1.1-373.g4e45e6b4.el9.x86_64 13/34 2024-09-22T03:04:44.732 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: libradosstriper1-2:19.1.1-373.g4e45e6b4.el9.x86_64 13/34 2024-09-22T03:04:44.767 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : thrift-0.15.0-2.el9.x86_64 14/34 2024-09-22T03:04:44.879 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : re2-1:20211101-20.el9.x86_64 15/34 2024-09-22T03:04:44.918 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 16/34 2024-09-22T03:04:44.960 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-09-22T03:04:45.000 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 18/34 2024-09-22T03:04:45.030 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 19/34 2024-09-22T03:04:45.073 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-09-22T03:04:45.414 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-09-22T03:04:45.458 INFO:teuthology.orchestra.run.smithi153.stdout: Upgrading : librbd1-2:19.1.1-373.g4e45e6b4.el9.x86_64 22/34 2024-09-22T03:04:45.549 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: librbd1-2:19.1.1-373.g4e45e6b4.el9.x86_64 22/34 2024-09-22T03:04:45.579 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-rbd-2:19.1.1-373.g4e45e6b4.el9.x86_64 23/34 2024-09-22T03:04:45.630 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-09-22T03:04:46.097 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-common-2:19.1.1-373.g4e45e6b4.el9.x86_64 29/34 2024-09-22T03:04:46.112 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 25/34 2024-09-22T03:04:46.136 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-common-2:19.1.1-373.g4e45e6b4.el9.x86_64 29/34 2024-09-22T03:04:46.143 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : librgw2-2:19.1.1-373.g4e45e6b4.el9.x86_64 26/34 2024-09-22T03:04:46.198 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: librgw2-2:19.1.1-373.g4e45e6b4.el9.x86_64 26/34 2024-09-22T03:04:46.276 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-rgw-2:19.1.1-373.g4e45e6b4.el9.x86_64 27/34 2024-09-22T03:04:46.304 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-ceph-common-2:19.1.1-373.g4e45e6b4.el9.x86 28/34 2024-09-22T03:04:46.705 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-common-2:19.1.1-373.g4e45e6b4.el9.x86_64 29/34 2024-09-22T03:04:46.748 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-base-2:19.1.1-373.g4e45e6b4.el9.x86_64 30/34 2024-09-22T03:04:46.830 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-base-2:19.1.1-373.g4e45e6b4.el9.x86_64 30/34 2024-09-22T03:04:46.830 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-22T03:04:46.830 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-22T03:04:46.830 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-22T03:04:46.874 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-selinux-2:19.1.1-373.g4e45e6b4.el9.x86_64 31/34 2024-09-22T03:04:48.571 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-common-2:19.1.1-373.g4e45e6b4.el9.x86_64 29/34 2024-09-22T03:04:48.608 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-common-2:19.1.1-373.g4e45e6b4.el9.x86_64 29/34 2024-09-22T03:04:49.202 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-common-2:19.1.1-373.g4e45e6b4.el9.x86_64 29/34 2024-09-22T03:04:49.253 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-base-2:19.1.1-373.g4e45e6b4.el9.x86_64 30/34 2024-09-22T03:04:49.353 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-base-2:19.1.1-373.g4e45e6b4.el9.x86_64 30/34 2024-09-22T03:04:49.354 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-22T03:04:49.354 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-22T03:04:49.354 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-22T03:04:49.391 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-selinux-2:19.1.1-373.g4e45e6b4.el9.x86_64 31/34 2024-09-22T03:04:56.828 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-selinux-2:19.1.1-373.g4e45e6b4.el9.x86_64 31/34 2024-09-22T03:04:56.828 INFO:teuthology.orchestra.run.smithi196.stdout:skipping the directory /sys 2024-09-22T03:04:56.828 INFO:teuthology.orchestra.run.smithi196.stdout:skipping the directory /proc 2024-09-22T03:04:56.828 INFO:teuthology.orchestra.run.smithi196.stdout:skipping the directory /mnt 2024-09-22T03:04:56.828 INFO:teuthology.orchestra.run.smithi196.stdout:skipping the directory /var/tmp 2024-09-22T03:04:56.829 INFO:teuthology.orchestra.run.smithi196.stdout:skipping the directory /home 2024-09-22T03:04:56.829 INFO:teuthology.orchestra.run.smithi196.stdout:skipping the directory /root 2024-09-22T03:04:56.829 INFO:teuthology.orchestra.run.smithi196.stdout:skipping the directory /tmp 2024-09-22T03:04:56.829 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-22T03:04:57.802 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-radosgw-2:19.1.1-373.g4e45e6b4.el9.x86_64 32/34 2024-09-22T03:04:57.827 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-radosgw-2:19.1.1-373.g4e45e6b4.el9.x86_64 32/34 2024-09-22T03:04:57.827 INFO:teuthology.orchestra.run.smithi196.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-22T03:04:57.827 INFO:teuthology.orchestra.run.smithi196.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-22T03:04:57.828 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-22T03:04:57.828 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-22T03:04:57.828 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-22T03:04:57.829 INFO:teuthology.orchestra.run.smithi196.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-09-22T03:04:57.896 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-09-22T03:04:57.896 INFO:teuthology.orchestra.run.smithi196.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-09-22T03:04:59.068 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-09-22T03:04:59.068 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-base-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/34 2024-09-22T03:04:59.068 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-common-2:19.1.1-373.g4e45e6b4.el9.x86_64 2/34 2024-09-22T03:04:59.068 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-radosgw-2:19.1.1-373.g4e45e6b4.el9.x86_64 3/34 2024-09-22T03:04:59.068 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-selinux-2:19.1.1-373.g4e45e6b4.el9.x86_64 4/34 2024-09-22T03:04:59.068 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : libcephfs2-2:19.1.1-373.g4e45e6b4.el9.x86_64 5/34 2024-09-22T03:04:59.068 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : libradosstriper1-2:19.1.1-373.g4e45e6b4.el9.x86_64 6/34 2024-09-22T03:04:59.068 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : librgw2-2:19.1.1-373.g4e45e6b4.el9.x86_64 7/34 2024-09-22T03:04:59.069 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-ceph-argparse-2:19.1.1-373.g4e45e6b4.el9.x 8/34 2024-09-22T03:04:59.069 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-ceph-common-2:19.1.1-373.g4e45e6b4.el9.x86 9/34 2024-09-22T03:04:59.069 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-cephfs-2:19.1.1-373.g4e45e6b4.el9.x86_64 10/34 2024-09-22T03:04:59.069 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-rados-2:19.1.1-373.g4e45e6b4.el9.x86_64 11/34 2024-09-22T03:04:59.069 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-rbd-2:19.1.1-373.g4e45e6b4.el9.x86_64 12/34 2024-09-22T03:04:59.069 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-rgw-2:19.1.1-373.g4e45e6b4.el9.x86_64 13/34 2024-09-22T03:04:59.069 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-09-22T03:04:59.069 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-09-22T03:04:59.069 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-09-22T03:04:59.069 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-09-22T03:04:59.069 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-09-22T03:04:59.069 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-09-22T03:04:59.069 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-09-22T03:04:59.069 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-09-22T03:04:59.070 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-09-22T03:04:59.070 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 23/34 2024-09-22T03:04:59.070 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 24/34 2024-09-22T03:04:59.070 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 25/34 2024-09-22T03:04:59.070 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-09-22T03:04:59.070 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-09-22T03:04:59.070 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 28/34 2024-09-22T03:04:59.070 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : re2-1:20211101-20.el9.x86_64 29/34 2024-09-22T03:04:59.070 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-09-22T03:04:59.070 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : librados2-2:19.1.1-373.g4e45e6b4.el9.x86_64 31/34 2024-09-22T03:04:59.070 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-09-22T03:04:59.070 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : librbd1-2:19.1.1-373.g4e45e6b4.el9.x86_64 33/34 2024-09-22T03:04:59.406 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-selinux-2:19.1.1-373.g4e45e6b4.el9.x86_64 31/34 2024-09-22T03:04:59.406 INFO:teuthology.orchestra.run.smithi153.stdout:skipping the directory /sys 2024-09-22T03:04:59.406 INFO:teuthology.orchestra.run.smithi153.stdout:skipping the directory /proc 2024-09-22T03:04:59.406 INFO:teuthology.orchestra.run.smithi153.stdout:skipping the directory /mnt 2024-09-22T03:04:59.407 INFO:teuthology.orchestra.run.smithi153.stdout:skipping the directory /var/tmp 2024-09-22T03:04:59.407 INFO:teuthology.orchestra.run.smithi153.stdout:skipping the directory /home 2024-09-22T03:04:59.407 INFO:teuthology.orchestra.run.smithi153.stdout:skipping the directory /root 2024-09-22T03:04:59.407 INFO:teuthology.orchestra.run.smithi153.stdout:skipping the directory /tmp 2024-09-22T03:04:59.407 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-22T03:05:00.195 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-09-22T03:05:00.195 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-22T03:05:00.196 INFO:teuthology.orchestra.run.smithi196.stdout:Upgraded: 2024-09-22T03:05:00.196 INFO:teuthology.orchestra.run.smithi196.stdout: librados2-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T03:05:00.196 INFO:teuthology.orchestra.run.smithi196.stdout: librbd1-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T03:05:00.196 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-09-22T03:05:00.196 INFO:teuthology.orchestra.run.smithi196.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-22T03:05:00.196 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-base-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T03:05:00.196 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-common-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T03:05:00.196 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-radosgw-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T03:05:00.196 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-selinux-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T03:05:00.196 INFO:teuthology.orchestra.run.smithi196.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-22T03:05:00.196 INFO:teuthology.orchestra.run.smithi196.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-22T03:05:00.196 INFO:teuthology.orchestra.run.smithi196.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-22T03:05:00.197 INFO:teuthology.orchestra.run.smithi196.stdout: libcephfs2-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T03:05:00.197 INFO:teuthology.orchestra.run.smithi196.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-22T03:05:00.197 INFO:teuthology.orchestra.run.smithi196.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-22T03:05:00.197 INFO:teuthology.orchestra.run.smithi196.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-22T03:05:00.197 INFO:teuthology.orchestra.run.smithi196.stdout: libradosstriper1-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T03:05:00.197 INFO:teuthology.orchestra.run.smithi196.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-22T03:05:00.197 INFO:teuthology.orchestra.run.smithi196.stdout: librgw2-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T03:05:00.197 INFO:teuthology.orchestra.run.smithi196.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-22T03:05:00.197 INFO:teuthology.orchestra.run.smithi196.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-22T03:05:00.197 INFO:teuthology.orchestra.run.smithi196.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-22T03:05:00.197 INFO:teuthology.orchestra.run.smithi196.stdout: python3-ceph-argparse-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T03:05:00.197 INFO:teuthology.orchestra.run.smithi196.stdout: python3-ceph-common-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T03:05:00.198 INFO:teuthology.orchestra.run.smithi196.stdout: python3-cephfs-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T03:05:00.198 INFO:teuthology.orchestra.run.smithi196.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-22T03:05:00.198 INFO:teuthology.orchestra.run.smithi196.stdout: python3-rados-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T03:05:00.198 INFO:teuthology.orchestra.run.smithi196.stdout: python3-rbd-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T03:05:00.198 INFO:teuthology.orchestra.run.smithi196.stdout: python3-rgw-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T03:05:00.198 INFO:teuthology.orchestra.run.smithi196.stdout: qatlib-24.02.0-1.el9.x86_64 2024-09-22T03:05:00.198 INFO:teuthology.orchestra.run.smithi196.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-09-22T03:05:00.198 INFO:teuthology.orchestra.run.smithi196.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-09-22T03:05:00.198 INFO:teuthology.orchestra.run.smithi196.stdout: re2-1:20211101-20.el9.x86_64 2024-09-22T03:05:00.198 INFO:teuthology.orchestra.run.smithi196.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-22T03:05:00.198 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-22T03:05:00.198 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-09-22T03:05:00.395 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-radosgw-2:19.1.1-373.g4e45e6b4.el9.x86_64 32/34 2024-09-22T03:05:00.421 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-radosgw-2:19.1.1-373.g4e45e6b4.el9.x86_64 32/34 2024-09-22T03:05:00.421 INFO:teuthology.orchestra.run.smithi153.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-22T03:05:00.421 INFO:teuthology.orchestra.run.smithi153.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-22T03:05:00.421 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-22T03:05:00.421 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-22T03:05:00.421 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-22T03:05:00.423 INFO:teuthology.orchestra.run.smithi153.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-09-22T03:05:00.506 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-09-22T03:05:00.506 INFO:teuthology.orchestra.run.smithi153.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-09-22T03:05:00.533 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install ceph-test 2024-09-22T03:05:01.110 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:00:30 ago on Sun 22 Sep 2024 03:04:31 AM UTC. 2024-09-22T03:05:01.241 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-09-22T03:05:01.242 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-09-22T03:05:01.242 INFO:teuthology.orchestra.run.smithi196.stdout: Package Arch Version Repository Size 2024-09-22T03:05:01.242 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-09-22T03:05:01.242 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-09-22T03:05:01.242 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-test x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 124 M 2024-09-22T03:05:01.242 INFO:teuthology.orchestra.run.smithi196.stdout:Installing dependencies: 2024-09-22T03:05:01.242 INFO:teuthology.orchestra.run.smithi196.stdout: libasan x86_64 11.5.0-2.el9 appstream 414 k 2024-09-22T03:05:01.242 INFO:teuthology.orchestra.run.smithi196.stdout: libcephsqlite x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 158 k 2024-09-22T03:05:01.242 INFO:teuthology.orchestra.run.smithi196.stdout: libubsan x86_64 11.5.0-2.el9 appstream 183 k 2024-09-22T03:05:01.242 INFO:teuthology.orchestra.run.smithi196.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-22T03:05:01.242 INFO:teuthology.orchestra.run.smithi196.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-22T03:05:01.242 INFO:teuthology.orchestra.run.smithi196.stdout: yaml-cpp x86_64 0.6.3-5.el9 epel 122 k 2024-09-22T03:05:01.242 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-22T03:05:01.243 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-09-22T03:05:01.243 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-09-22T03:05:01.243 INFO:teuthology.orchestra.run.smithi196.stdout:Install 7 Packages 2024-09-22T03:05:01.243 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-22T03:05:01.243 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 125 M 2024-09-22T03:05:01.243 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 1.1 G 2024-09-22T03:05:01.243 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-09-22T03:05:01.651 INFO:teuthology.orchestra.run.smithi196.stdout:(1/7): libasan-11.5.0-2.el9.x86_64.rpm 3.2 MB/s | 414 kB 00:00 2024-09-22T03:05:01.679 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-09-22T03:05:01.679 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-base-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/34 2024-09-22T03:05:01.679 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-common-2:19.1.1-373.g4e45e6b4.el9.x86_64 2/34 2024-09-22T03:05:01.680 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-radosgw-2:19.1.1-373.g4e45e6b4.el9.x86_64 3/34 2024-09-22T03:05:01.680 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-selinux-2:19.1.1-373.g4e45e6b4.el9.x86_64 4/34 2024-09-22T03:05:01.680 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libcephfs2-2:19.1.1-373.g4e45e6b4.el9.x86_64 5/34 2024-09-22T03:05:01.680 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libradosstriper1-2:19.1.1-373.g4e45e6b4.el9.x86_64 6/34 2024-09-22T03:05:01.680 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : librgw2-2:19.1.1-373.g4e45e6b4.el9.x86_64 7/34 2024-09-22T03:05:01.680 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-ceph-argparse-2:19.1.1-373.g4e45e6b4.el9.x 8/34 2024-09-22T03:05:01.680 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-ceph-common-2:19.1.1-373.g4e45e6b4.el9.x86 9/34 2024-09-22T03:05:01.680 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-cephfs-2:19.1.1-373.g4e45e6b4.el9.x86_64 10/34 2024-09-22T03:05:01.680 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-rados-2:19.1.1-373.g4e45e6b4.el9.x86_64 11/34 2024-09-22T03:05:01.680 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-rbd-2:19.1.1-373.g4e45e6b4.el9.x86_64 12/34 2024-09-22T03:05:01.680 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-rgw-2:19.1.1-373.g4e45e6b4.el9.x86_64 13/34 2024-09-22T03:05:01.680 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-09-22T03:05:01.681 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-09-22T03:05:01.681 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-09-22T03:05:01.681 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-09-22T03:05:01.681 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-09-22T03:05:01.681 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-09-22T03:05:01.681 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-09-22T03:05:01.681 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-09-22T03:05:01.681 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-09-22T03:05:01.681 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 23/34 2024-09-22T03:05:01.681 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 24/34 2024-09-22T03:05:01.681 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 25/34 2024-09-22T03:05:01.681 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-09-22T03:05:01.681 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-09-22T03:05:01.682 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 28/34 2024-09-22T03:05:01.682 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : re2-1:20211101-20.el9.x86_64 29/34 2024-09-22T03:05:01.682 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-09-22T03:05:01.683 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : librados2-2:19.1.1-373.g4e45e6b4.el9.x86_64 31/34 2024-09-22T03:05:01.683 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-09-22T03:05:01.683 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : librbd1-2:19.1.1-373.g4e45e6b4.el9.x86_64 33/34 2024-09-22T03:05:01.692 INFO:teuthology.orchestra.run.smithi196.stdout:(2/7): libubsan-11.5.0-2.el9.x86_64.rpm 4.3 MB/s | 183 kB 00:00 2024-09-22T03:05:01.734 INFO:teuthology.orchestra.run.smithi196.stdout:(3/7): socat-1.7.4.1-6.el9.x86_64.rpm 7.1 MB/s | 304 kB 00:00 2024-09-22T03:05:01.760 INFO:teuthology.orchestra.run.smithi196.stdout:(4/7): libcephsqlite-19.1.1-373.g4e45e6b4.el9.x 669 kB/s | 158 kB 00:00 2024-09-22T03:05:01.776 INFO:teuthology.orchestra.run.smithi196.stdout:(5/7): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.5 MB/s | 64 kB 00:00 2024-09-22T03:05:01.819 INFO:teuthology.orchestra.run.smithi196.stdout:(6/7): yaml-cpp-0.6.3-5.el9.x86_64.rpm 2.0 MB/s | 122 kB 00:00 2024-09-22T03:05:02.695 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-09-22T03:05:02.695 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-22T03:05:02.695 INFO:teuthology.orchestra.run.smithi153.stdout:Upgraded: 2024-09-22T03:05:02.695 INFO:teuthology.orchestra.run.smithi153.stdout: librados2-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T03:05:02.695 INFO:teuthology.orchestra.run.smithi153.stdout: librbd1-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T03:05:02.696 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-09-22T03:05:02.696 INFO:teuthology.orchestra.run.smithi153.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-22T03:05:02.696 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-base-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T03:05:02.696 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-common-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T03:05:02.696 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-radosgw-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T03:05:02.696 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-selinux-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T03:05:02.696 INFO:teuthology.orchestra.run.smithi153.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-22T03:05:02.696 INFO:teuthology.orchestra.run.smithi153.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-22T03:05:02.696 INFO:teuthology.orchestra.run.smithi153.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-22T03:05:02.696 INFO:teuthology.orchestra.run.smithi153.stdout: libcephfs2-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T03:05:02.696 INFO:teuthology.orchestra.run.smithi153.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-22T03:05:02.696 INFO:teuthology.orchestra.run.smithi153.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-22T03:05:02.696 INFO:teuthology.orchestra.run.smithi153.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-22T03:05:02.696 INFO:teuthology.orchestra.run.smithi153.stdout: libradosstriper1-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T03:05:02.697 INFO:teuthology.orchestra.run.smithi153.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-22T03:05:02.697 INFO:teuthology.orchestra.run.smithi153.stdout: librgw2-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T03:05:02.697 INFO:teuthology.orchestra.run.smithi153.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-22T03:05:02.697 INFO:teuthology.orchestra.run.smithi153.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-22T03:05:02.697 INFO:teuthology.orchestra.run.smithi153.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-22T03:05:02.697 INFO:teuthology.orchestra.run.smithi153.stdout: python3-ceph-argparse-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T03:05:02.697 INFO:teuthology.orchestra.run.smithi153.stdout: python3-ceph-common-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T03:05:02.697 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cephfs-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T03:05:02.697 INFO:teuthology.orchestra.run.smithi153.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-22T03:05:02.697 INFO:teuthology.orchestra.run.smithi153.stdout: python3-rados-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T03:05:02.697 INFO:teuthology.orchestra.run.smithi153.stdout: python3-rbd-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T03:05:02.697 INFO:teuthology.orchestra.run.smithi153.stdout: python3-rgw-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T03:05:02.697 INFO:teuthology.orchestra.run.smithi153.stdout: qatlib-24.02.0-1.el9.x86_64 2024-09-22T03:05:02.697 INFO:teuthology.orchestra.run.smithi153.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-09-22T03:05:02.697 INFO:teuthology.orchestra.run.smithi153.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-09-22T03:05:02.698 INFO:teuthology.orchestra.run.smithi153.stdout: re2-1:20211101-20.el9.x86_64 2024-09-22T03:05:02.698 INFO:teuthology.orchestra.run.smithi153.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-22T03:05:02.698 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-22T03:05:02.698 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-09-22T03:05:03.035 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install ceph-test 2024-09-22T03:05:03.614 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:00:31 ago on Sun 22 Sep 2024 03:04:32 AM UTC. 2024-09-22T03:05:03.751 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-09-22T03:05:03.751 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-09-22T03:05:03.752 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repository Size 2024-09-22T03:05:03.752 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-09-22T03:05:03.752 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-09-22T03:05:03.752 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-test x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 124 M 2024-09-22T03:05:03.752 INFO:teuthology.orchestra.run.smithi153.stdout:Installing dependencies: 2024-09-22T03:05:03.752 INFO:teuthology.orchestra.run.smithi153.stdout: libasan x86_64 11.5.0-2.el9 appstream 414 k 2024-09-22T03:05:03.752 INFO:teuthology.orchestra.run.smithi153.stdout: libcephsqlite x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 158 k 2024-09-22T03:05:03.752 INFO:teuthology.orchestra.run.smithi153.stdout: libubsan x86_64 11.5.0-2.el9 appstream 183 k 2024-09-22T03:05:03.752 INFO:teuthology.orchestra.run.smithi153.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-22T03:05:03.752 INFO:teuthology.orchestra.run.smithi153.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-22T03:05:03.752 INFO:teuthology.orchestra.run.smithi153.stdout: yaml-cpp x86_64 0.6.3-5.el9 epel 122 k 2024-09-22T03:05:03.752 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-22T03:05:03.753 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-09-22T03:05:03.753 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-09-22T03:05:03.753 INFO:teuthology.orchestra.run.smithi153.stdout:Install 7 Packages 2024-09-22T03:05:03.753 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-22T03:05:03.753 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 125 M 2024-09-22T03:05:03.753 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 1.1 G 2024-09-22T03:05:03.753 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-09-22T03:05:04.193 INFO:teuthology.orchestra.run.smithi153.stdout:(1/7): libasan-11.5.0-2.el9.x86_64.rpm 2.9 MB/s | 414 kB 00:00 2024-09-22T03:05:04.235 INFO:teuthology.orchestra.run.smithi153.stdout:(2/7): libubsan-11.5.0-2.el9.x86_64.rpm 4.3 MB/s | 183 kB 00:00 2024-09-22T03:05:04.277 INFO:teuthology.orchestra.run.smithi153.stdout:(3/7): socat-1.7.4.1-6.el9.x86_64.rpm 7.2 MB/s | 304 kB 00:00 2024-09-22T03:05:04.302 INFO:teuthology.orchestra.run.smithi153.stdout:(4/7): libcephsqlite-19.1.1-373.g4e45e6b4.el9.x 630 kB/s | 158 kB 00:00 2024-09-22T03:05:04.319 INFO:teuthology.orchestra.run.smithi153.stdout:(5/7): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.5 MB/s | 64 kB 00:00 2024-09-22T03:05:04.361 INFO:teuthology.orchestra.run.smithi153.stdout:(6/7): yaml-cpp-0.6.3-5.el9.x86_64.rpm 2.0 MB/s | 122 kB 00:00 2024-09-22T03:05:04.977 INFO:teuthology.orchestra.run.smithi196.stdout:(7/7): ceph-test-19.1.1-373.g4e45e6b4.el9.x86_6 36 MB/s | 124 MB 00:03 2024-09-22T03:05:04.980 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-09-22T03:05:04.981 INFO:teuthology.orchestra.run.smithi196.stdout:Total 34 MB/s | 125 MB 00:03 2024-09-22T03:05:05.035 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-09-22T03:05:05.056 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-09-22T03:05:05.056 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-09-22T03:05:05.883 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-09-22T03:05:05.883 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-09-22T03:05:06.951 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-09-22T03:05:07.017 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : yaml-cpp-0.6.3-5.el9.x86_64 1/7 2024-09-22T03:05:07.067 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 2/7 2024-09-22T03:05:07.107 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 3/7 2024-09-22T03:05:07.136 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : libubsan-11.5.0-2.el9.x86_64 4/7 2024-09-22T03:05:07.178 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : libasan-11.5.0-2.el9.x86_64 5/7 2024-09-22T03:05:07.205 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : libcephsqlite-2:19.1.1-373.g4e45e6b4.el9.x86_64 6/7 2024-09-22T03:05:07.478 INFO:teuthology.orchestra.run.smithi153.stdout:(7/7): ceph-test-19.1.1-373.g4e45e6b4.el9.x86_6 36 MB/s | 124 MB 00:03 2024-09-22T03:05:07.481 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-09-22T03:05:07.481 INFO:teuthology.orchestra.run.smithi153.stdout:Total 34 MB/s | 125 MB 00:03 2024-09-22T03:05:07.533 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-09-22T03:05:07.554 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-09-22T03:05:07.554 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-09-22T03:05:08.374 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-09-22T03:05:08.375 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-09-22T03:05:09.426 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-09-22T03:05:09.510 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : yaml-cpp-0.6.3-5.el9.x86_64 1/7 2024-09-22T03:05:09.554 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 2/7 2024-09-22T03:05:09.596 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 3/7 2024-09-22T03:05:09.629 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libubsan-11.5.0-2.el9.x86_64 4/7 2024-09-22T03:05:09.670 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libasan-11.5.0-2.el9.x86_64 5/7 2024-09-22T03:05:09.698 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libcephsqlite-2:19.1.1-373.g4e45e6b4.el9.x86_64 6/7 2024-09-22T03:05:24.416 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: libcephsqlite-2:19.1.1-373.g4e45e6b4.el9.x86_64 6/7 2024-09-22T03:05:24.486 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-test-2:19.1.1-373.g4e45e6b4.el9.x86_64 7/7 2024-09-22T03:05:26.884 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: libcephsqlite-2:19.1.1-373.g4e45e6b4.el9.x86_64 6/7 2024-09-22T03:05:26.961 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-test-2:19.1.1-373.g4e45e6b4.el9.x86_64 7/7 2024-09-22T03:05:32.681 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-test-2:19.1.1-373.g4e45e6b4.el9.x86_64 7/7 2024-09-22T03:05:32.681 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-test-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/7 2024-09-22T03:05:32.681 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : libcephsqlite-2:19.1.1-373.g4e45e6b4.el9.x86_64 2/7 2024-09-22T03:05:32.681 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : libasan-11.5.0-2.el9.x86_64 3/7 2024-09-22T03:05:32.681 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : libubsan-11.5.0-2.el9.x86_64 4/7 2024-09-22T03:05:32.681 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 5/7 2024-09-22T03:05:32.681 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 6/7 2024-09-22T03:05:33.070 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : yaml-cpp-0.6.3-5.el9.x86_64 7/7 2024-09-22T03:05:33.071 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-22T03:05:33.071 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-09-22T03:05:33.071 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-test-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T03:05:33.071 INFO:teuthology.orchestra.run.smithi196.stdout: libasan-11.5.0-2.el9.x86_64 2024-09-22T03:05:33.071 INFO:teuthology.orchestra.run.smithi196.stdout: libcephsqlite-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T03:05:33.071 INFO:teuthology.orchestra.run.smithi196.stdout: libubsan-11.5.0-2.el9.x86_64 2024-09-22T03:05:33.071 INFO:teuthology.orchestra.run.smithi196.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-22T03:05:33.071 INFO:teuthology.orchestra.run.smithi196.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-22T03:05:33.071 INFO:teuthology.orchestra.run.smithi196.stdout: yaml-cpp-0.6.3-5.el9.x86_64 2024-09-22T03:05:33.072 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-22T03:05:33.072 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-09-22T03:05:33.380 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install ceph 2024-09-22T03:05:33.961 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:01:02 ago on Sun 22 Sep 2024 03:04:31 AM UTC. 2024-09-22T03:05:34.095 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-09-22T03:05:34.097 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-09-22T03:05:34.097 INFO:teuthology.orchestra.run.smithi196.stdout: Package Arch Version Repository Size 2024-09-22T03:05:34.097 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-09-22T03:05:34.097 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-09-22T03:05:34.098 INFO:teuthology.orchestra.run.smithi196.stdout: ceph x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 6.6 k 2024-09-22T03:05:34.098 INFO:teuthology.orchestra.run.smithi196.stdout:Installing dependencies: 2024-09-22T03:05:34.098 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mds x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 2.3 M 2024-09-22T03:05:34.098 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mgr x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 1.5 M 2024-09-22T03:05:34.098 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mgr-modules-core noarch 2:19.1.1-373.g4e45e6b4.el9 ceph-noarch 249 k 2024-09-22T03:05:34.098 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mon x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 4.9 M 2024-09-22T03:05:34.098 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-osd x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 120 M 2024-09-22T03:05:34.098 INFO:teuthology.orchestra.run.smithi196.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-09-22T03:05:34.098 INFO:teuthology.orchestra.run.smithi196.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-09-22T03:05:34.098 INFO:teuthology.orchestra.run.smithi196.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-09-22T03:05:34.098 INFO:teuthology.orchestra.run.smithi196.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-09-22T03:05:34.098 INFO:teuthology.orchestra.run.smithi196.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-22T03:05:34.098 INFO:teuthology.orchestra.run.smithi196.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-22T03:05:34.099 INFO:teuthology.orchestra.run.smithi196.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-22T03:05:34.099 INFO:teuthology.orchestra.run.smithi196.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-22T03:05:34.099 INFO:teuthology.orchestra.run.smithi196.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-22T03:05:34.099 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-22T03:05:34.099 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-22T03:05:34.099 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-09-22T03:05:34.099 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-22T03:05:34.099 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-22T03:05:34.099 INFO:teuthology.orchestra.run.smithi196.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-22T03:05:34.099 INFO:teuthology.orchestra.run.smithi196.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-22T03:05:34.099 INFO:teuthology.orchestra.run.smithi196.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-22T03:05:34.099 INFO:teuthology.orchestra.run.smithi196.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-22T03:05:34.099 INFO:teuthology.orchestra.run.smithi196.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-22T03:05:34.100 INFO:teuthology.orchestra.run.smithi196.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-22T03:05:34.100 INFO:teuthology.orchestra.run.smithi196.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-22T03:05:34.100 INFO:teuthology.orchestra.run.smithi196.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-22T03:05:34.100 INFO:teuthology.orchestra.run.smithi196.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-22T03:05:34.100 INFO:teuthology.orchestra.run.smithi196.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-22T03:05:34.100 INFO:teuthology.orchestra.run.smithi196.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-22T03:05:34.100 INFO:teuthology.orchestra.run.smithi196.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-22T03:05:34.100 INFO:teuthology.orchestra.run.smithi196.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-22T03:05:34.100 INFO:teuthology.orchestra.run.smithi196.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-22T03:05:34.100 INFO:teuthology.orchestra.run.smithi196.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-22T03:05:34.100 INFO:teuthology.orchestra.run.smithi196.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-22T03:05:34.100 INFO:teuthology.orchestra.run.smithi196.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-22T03:05:34.100 INFO:teuthology.orchestra.run.smithi196.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-22T03:05:34.100 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-22T03:05:34.101 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-09-22T03:05:34.101 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-09-22T03:05:34.101 INFO:teuthology.orchestra.run.smithi196.stdout:Install 38 Packages 2024-09-22T03:05:34.101 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-22T03:05:34.101 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 134 M 2024-09-22T03:05:34.101 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 1.4 G 2024-09-22T03:05:34.102 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-09-22T03:05:34.844 INFO:teuthology.orchestra.run.smithi196.stdout:(1/38): ceph-19.1.1-373.g4e45e6b4.el9.x86_64.rp 52 kB/s | 6.6 kB 00:00 2024-09-22T03:05:35.036 INFO:teuthology.orchestra.run.smithi196.stdout:(2/38): ceph-mds-19.1.1-373.g4e45e6b4.el9.x86_6 7.1 MB/s | 2.3 MB 00:00 2024-09-22T03:05:35.069 INFO:teuthology.orchestra.run.smithi196.stdout:(3/38): ceph-mgr-19.1.1-373.g4e45e6b4.el9.x86_6 4.3 MB/s | 1.5 MB 00:00 2024-09-22T03:05:35.124 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-test-2:19.1.1-373.g4e45e6b4.el9.x86_64 7/7 2024-09-22T03:05:35.124 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-test-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/7 2024-09-22T03:05:35.124 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libcephsqlite-2:19.1.1-373.g4e45e6b4.el9.x86_64 2/7 2024-09-22T03:05:35.124 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libasan-11.5.0-2.el9.x86_64 3/7 2024-09-22T03:05:35.124 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libubsan-11.5.0-2.el9.x86_64 4/7 2024-09-22T03:05:35.124 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 5/7 2024-09-22T03:05:35.125 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 6/7 2024-09-22T03:05:35.161 INFO:teuthology.orchestra.run.smithi196.stdout:(4/38): ceph-mgr-modules-core-19.1.1-373.g4e45e 2.7 MB/s | 249 kB 00:00 2024-09-22T03:05:35.295 INFO:teuthology.orchestra.run.smithi196.stdout:(5/38): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.9 MB/s | 253 kB 00:00 2024-09-22T03:05:35.379 INFO:teuthology.orchestra.run.smithi196.stdout:(6/38): ceph-mon-19.1.1-373.g4e45e6b4.el9.x86_6 9.1 MB/s | 4.9 MB 00:00 2024-09-22T03:05:35.429 INFO:teuthology.orchestra.run.smithi196.stdout:(7/38): python3-cryptography-36.0.1-4.el9.x86_6 9.2 MB/s | 1.2 MB 00:00 2024-09-22T03:05:35.496 INFO:teuthology.orchestra.run.smithi196.stdout:(8/38): python3-ply-3.11-14.el9.noarch.rpm 912 kB/s | 106 kB 00:00 2024-09-22T03:05:35.521 INFO:teuthology.orchestra.run.smithi196.stdout:(9/38): python3-pycparser-2.20-6.el9.noarch.rpm 1.4 MB/s | 135 kB 00:00 2024-09-22T03:05:35.536 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : yaml-cpp-0.6.3-5.el9.x86_64 7/7 2024-09-22T03:05:35.536 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-22T03:05:35.536 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-09-22T03:05:35.536 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-test-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T03:05:35.540 INFO:teuthology.orchestra.run.smithi153.stdout: libasan-11.5.0-2.el9.x86_64 2024-09-22T03:05:35.540 INFO:teuthology.orchestra.run.smithi153.stdout: libcephsqlite-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T03:05:35.540 INFO:teuthology.orchestra.run.smithi153.stdout: libubsan-11.5.0-2.el9.x86_64 2024-09-22T03:05:35.540 INFO:teuthology.orchestra.run.smithi153.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-22T03:05:35.540 INFO:teuthology.orchestra.run.smithi153.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-22T03:05:35.541 INFO:teuthology.orchestra.run.smithi153.stdout: yaml-cpp-0.6.3-5.el9.x86_64 2024-09-22T03:05:35.541 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-22T03:05:35.541 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-09-22T03:05:35.588 INFO:teuthology.orchestra.run.smithi196.stdout:(10/38): python3-requests-2.25.1-8.el9.noarch.r 1.3 MB/s | 125 kB 00:00 2024-09-22T03:05:35.613 INFO:teuthology.orchestra.run.smithi196.stdout:(11/38): python3-urllib3-1.26.5-6.el9.noarch.rp 2.3 MB/s | 215 kB 00:00 2024-09-22T03:05:35.846 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install ceph 2024-09-22T03:05:36.389 INFO:teuthology.orchestra.run.smithi196.stdout:(12/38): lua-5.4.4-4.el9.x86_64.rpm 235 kB/s | 188 kB 00:00 2024-09-22T03:05:36.431 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:04 ago on Sun 22 Sep 2024 03:04:32 AM UTC. 2024-09-22T03:05:36.569 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-09-22T03:05:36.571 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-09-22T03:05:36.571 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repository Size 2024-09-22T03:05:36.571 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-09-22T03:05:36.571 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-09-22T03:05:36.571 INFO:teuthology.orchestra.run.smithi153.stdout: ceph x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 6.6 k 2024-09-22T03:05:36.571 INFO:teuthology.orchestra.run.smithi153.stdout:Installing dependencies: 2024-09-22T03:05:36.571 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mds x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 2.3 M 2024-09-22T03:05:36.571 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 1.5 M 2024-09-22T03:05:36.571 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-modules-core noarch 2:19.1.1-373.g4e45e6b4.el9 ceph-noarch 249 k 2024-09-22T03:05:36.572 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mon x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 4.9 M 2024-09-22T03:05:36.572 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-osd x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 120 M 2024-09-22T03:05:36.572 INFO:teuthology.orchestra.run.smithi153.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-09-22T03:05:36.572 INFO:teuthology.orchestra.run.smithi153.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-09-22T03:05:36.572 INFO:teuthology.orchestra.run.smithi153.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-09-22T03:05:36.572 INFO:teuthology.orchestra.run.smithi153.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-09-22T03:05:36.572 INFO:teuthology.orchestra.run.smithi153.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-22T03:05:36.572 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-22T03:05:36.572 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-22T03:05:36.572 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-22T03:05:36.572 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-22T03:05:36.572 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-22T03:05:36.572 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-22T03:05:36.573 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-09-22T03:05:36.573 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-22T03:05:36.573 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-22T03:05:36.573 INFO:teuthology.orchestra.run.smithi153.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-22T03:05:36.573 INFO:teuthology.orchestra.run.smithi153.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-22T03:05:36.573 INFO:teuthology.orchestra.run.smithi153.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-22T03:05:36.573 INFO:teuthology.orchestra.run.smithi153.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-22T03:05:36.573 INFO:teuthology.orchestra.run.smithi153.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-22T03:05:36.573 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-22T03:05:36.573 INFO:teuthology.orchestra.run.smithi153.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-22T03:05:36.573 INFO:teuthology.orchestra.run.smithi153.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-22T03:05:36.573 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-22T03:05:36.573 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-22T03:05:36.573 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-22T03:05:36.573 INFO:teuthology.orchestra.run.smithi153.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-22T03:05:36.574 INFO:teuthology.orchestra.run.smithi153.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-22T03:05:36.574 INFO:teuthology.orchestra.run.smithi153.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-22T03:05:36.574 INFO:teuthology.orchestra.run.smithi153.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-22T03:05:36.574 INFO:teuthology.orchestra.run.smithi153.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-22T03:05:36.574 INFO:teuthology.orchestra.run.smithi153.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-22T03:05:36.574 INFO:teuthology.orchestra.run.smithi153.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-22T03:05:36.574 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-22T03:05:36.574 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-09-22T03:05:36.574 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-09-22T03:05:36.574 INFO:teuthology.orchestra.run.smithi153.stdout:Install 38 Packages 2024-09-22T03:05:36.574 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-22T03:05:36.575 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 134 M 2024-09-22T03:05:36.575 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 1.4 G 2024-09-22T03:05:36.575 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-09-22T03:05:36.597 INFO:teuthology.orchestra.run.smithi196.stdout:(13/38): protobuf-3.14.0-13.el9.x86_64.rpm 1.0 MB/s | 1.0 MB 00:00 2024-09-22T03:05:37.097 INFO:teuthology.orchestra.run.smithi196.stdout:(14/38): python3-mako-1.1.4-6.el9.noarch.rpm 243 kB/s | 172 kB 00:00 2024-09-22T03:05:37.148 INFO:teuthology.orchestra.run.smithi196.stdout:(15/38): python3-markupsafe-1.1.1-12.el9.x86_64 63 kB/s | 35 kB 00:00 2024-09-22T03:05:37.186 INFO:teuthology.orchestra.run.smithi153.stdout:(1/38): ceph-19.1.1-373.g4e45e6b4.el9.x86_64.rp 26 kB/s | 6.6 kB 00:00 2024-09-22T03:05:37.611 INFO:teuthology.orchestra.run.smithi153.stdout:(2/38): ceph-mds-19.1.1-373.g4e45e6b4.el9.x86_6 3.3 MB/s | 2.3 MB 00:00 2024-09-22T03:05:37.645 INFO:teuthology.orchestra.run.smithi153.stdout:(3/38): ceph-mgr-19.1.1-373.g4e45e6b4.el9.x86_6 2.1 MB/s | 1.5 MB 00:00 2024-09-22T03:05:37.715 INFO:teuthology.orchestra.run.smithi153.stdout:(4/38): ceph-mgr-modules-core-19.1.1-373.g4e45e 3.5 MB/s | 249 kB 00:00 2024-09-22T03:05:37.849 INFO:teuthology.orchestra.run.smithi153.stdout:(5/38): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.9 MB/s | 253 kB 00:00 2024-09-22T03:05:37.957 INFO:teuthology.orchestra.run.smithi153.stdout:(6/38): python3-cryptography-36.0.1-4.el9.x86_6 11 MB/s | 1.2 MB 00:00 2024-09-22T03:05:38.033 INFO:teuthology.orchestra.run.smithi153.stdout:(7/38): ceph-mon-19.1.1-373.g4e45e6b4.el9.x86_6 5.8 MB/s | 4.9 MB 00:00 2024-09-22T03:05:38.058 INFO:teuthology.orchestra.run.smithi153.stdout:(8/38): python3-ply-3.11-14.el9.noarch.rpm 1.0 MB/s | 106 kB 00:00 2024-09-22T03:05:38.100 INFO:teuthology.orchestra.run.smithi153.stdout:(9/38): python3-requests-2.25.1-8.el9.noarch.rp 2.9 MB/s | 125 kB 00:00 2024-09-22T03:05:38.150 INFO:teuthology.orchestra.run.smithi153.stdout:(10/38): python3-urllib3-1.26.5-6.el9.noarch.rp 4.2 MB/s | 215 kB 00:00 2024-09-22T03:05:38.175 INFO:teuthology.orchestra.run.smithi153.stdout:(11/38): python3-pycparser-2.20-6.el9.noarch.rp 950 kB/s | 135 kB 00:00 2024-09-22T03:05:38.206 INFO:teuthology.orchestra.run.smithi196.stdout:(16/38): ceph-osd-19.1.1-373.g4e45e6b4.el9.x86_ 38 MB/s | 120 MB 00:03 2024-09-22T03:05:38.375 INFO:teuthology.orchestra.run.smithi153.stdout:(12/38): lua-5.4.4-4.el9.x86_64.rpm 837 kB/s | 188 kB 00:00 2024-09-22T03:05:38.434 INFO:teuthology.orchestra.run.smithi153.stdout:(13/38): python3-mako-1.1.4-6.el9.noarch.rpm 2.9 MB/s | 172 kB 00:00 2024-09-22T03:05:38.493 INFO:teuthology.orchestra.run.smithi153.stdout:(14/38): python3-markupsafe-1.1.1-12.el9.x86_64 594 kB/s | 35 kB 00:00 2024-09-22T03:05:38.527 INFO:teuthology.orchestra.run.smithi153.stdout:(15/38): protobuf-3.14.0-13.el9.x86_64.rpm 2.9 MB/s | 1.0 MB 00:00 2024-09-22T03:05:38.552 INFO:teuthology.orchestra.run.smithi153.stdout:(16/38): python3-packaging-20.9-5.el9.noarch.rp 1.3 MB/s | 77 kB 00:00 2024-09-22T03:05:38.577 INFO:teuthology.orchestra.run.smithi153.stdout:(17/38): python3-pytz-2021.1-5.el9.noarch.rpm 1.0 MB/s | 51 kB 00:00 2024-09-22T03:05:38.602 INFO:teuthology.orchestra.run.smithi153.stdout:(18/38): python3-toml-0.10.2-6.el9.noarch.rpm 828 kB/s | 42 kB 00:00 2024-09-22T03:05:38.653 INFO:teuthology.orchestra.run.smithi153.stdout:(19/38): luarocks-3.9.2-5.el9.noarch.rpm 3.0 MB/s | 151 kB 00:00 2024-09-22T03:05:38.678 INFO:teuthology.orchestra.run.smithi153.stdout:(20/38): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 1.7 MB/s | 43 kB 00:00 2024-09-22T03:05:38.685 INFO:teuthology.orchestra.run.smithi196.stdout:(17/38): python3-pytz-2021.1-5.el9.noarch.rpm 33 kB/s | 51 kB 00:01 2024-09-22T03:05:38.710 INFO:teuthology.orchestra.run.smithi196.stdout:(18/38): python3-packaging-20.9-5.el9.noarch.rp 48 kB/s | 77 kB 00:01 2024-09-22T03:05:38.711 INFO:teuthology.orchestra.run.smithi153.stdout:(21/38): python3-cheroot-8.6.0-4.el9.noarch.rpm 5.0 MB/s | 172 kB 00:00 2024-09-22T03:05:38.745 INFO:teuthology.orchestra.run.smithi153.stdout:(22/38): python3-cherrypy-18.6.1-2.el9.noarch.r 11 MB/s | 358 kB 00:00 2024-09-22T03:05:38.769 INFO:teuthology.orchestra.run.smithi196.stdout:(19/38): luarocks-3.9.2-5.el9.noarch.rpm 2.5 MB/s | 151 kB 00:00 2024-09-22T03:05:38.770 INFO:teuthology.orchestra.run.smithi153.stdout:(23/38): python3-jaraco-8.2.1-3.el9.noarch.rpm 433 kB/s | 11 kB 00:00 2024-09-22T03:05:38.794 INFO:teuthology.orchestra.run.smithi196.stdout:(20/38): python3-toml-0.10.2-6.el9.noarch.rpm 71 kB/s | 42 kB 00:00 2024-09-22T03:05:38.795 INFO:teuthology.orchestra.run.smithi153.stdout:(24/38): python3-jaraco-classes-3.2.1-5.el9.noa 706 kB/s | 18 kB 00:00 2024-09-22T03:05:38.819 INFO:teuthology.orchestra.run.smithi196.stdout:(21/38): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 878 kB/s | 43 kB 00:00 2024-09-22T03:05:38.821 INFO:teuthology.orchestra.run.smithi153.stdout:(25/38): python3-jaraco-collections-3.0.0-8.el9 926 kB/s | 23 kB 00:00 2024-09-22T03:05:38.854 INFO:teuthology.orchestra.run.smithi153.stdout:(26/38): python3-jaraco-functools-3.5.0-2.el9.n 738 kB/s | 19 kB 00:00 2024-09-22T03:05:38.861 INFO:teuthology.orchestra.run.smithi196.stdout:(22/38): python3-cherrypy-18.6.1-2.el9.noarch.r 10 MB/s | 358 kB 00:00 2024-09-22T03:05:38.880 INFO:teuthology.orchestra.run.smithi153.stdout:(27/38): python3-jaraco-text-3.2.0-6.el9.noarch 789 kB/s | 20 kB 00:00 2024-09-22T03:05:38.886 INFO:teuthology.orchestra.run.smithi196.stdout:(23/38): python3-jaraco-8.2.1-3.el9.noarch.rpm 427 kB/s | 11 kB 00:00 2024-09-22T03:05:38.905 INFO:teuthology.orchestra.run.smithi153.stdout:(28/38): python3-logutils-0.3.5-21.el9.noarch.r 1.8 MB/s | 46 kB 00:00 2024-09-22T03:05:38.911 INFO:teuthology.orchestra.run.smithi196.stdout:(24/38): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.4 MB/s | 172 kB 00:00 2024-09-22T03:05:38.930 INFO:teuthology.orchestra.run.smithi153.stdout:(29/38): python3-more-itertools-8.12.0-2.el9.no 3.2 MB/s | 79 kB 00:00 2024-09-22T03:05:38.937 INFO:teuthology.orchestra.run.smithi196.stdout:(25/38): python3-jaraco-classes-3.2.1-5.el9.noa 352 kB/s | 18 kB 00:00 2024-09-22T03:05:38.962 INFO:teuthology.orchestra.run.smithi196.stdout:(26/38): python3-jaraco-functools-3.5.0-2.el9.n 788 kB/s | 19 kB 00:00 2024-09-22T03:05:38.972 INFO:teuthology.orchestra.run.smithi153.stdout:(30/38): python3-pecan-1.4.2-3.el9.noarch.rpm 6.4 MB/s | 272 kB 00:00 2024-09-22T03:05:38.987 INFO:teuthology.orchestra.run.smithi196.stdout:(27/38): python3-jaraco-collections-3.0.0-8.el9 463 kB/s | 23 kB 00:00 2024-09-22T03:05:38.997 INFO:teuthology.orchestra.run.smithi153.stdout:(31/38): python3-portend-3.1.0-2.el9.noarch.rpm 665 kB/s | 16 kB 00:00 2024-09-22T03:05:39.013 INFO:teuthology.orchestra.run.smithi196.stdout:(28/38): lua-devel-5.4.4-4.el9.x86_64.rpm 68 kB/s | 22 kB 00:00 2024-09-22T03:05:39.023 INFO:teuthology.orchestra.run.smithi153.stdout:(32/38): python3-pyOpenSSL-21.0.0-1.el9.noarch. 3.5 MB/s | 90 kB 00:00 2024-09-22T03:05:39.038 INFO:teuthology.orchestra.run.smithi196.stdout:(29/38): python3-jaraco-text-3.2.0-6.el9.noarch 260 kB/s | 20 kB 00:00 2024-09-22T03:05:39.048 INFO:teuthology.orchestra.run.smithi153.stdout:(33/38): python3-tempora-5.0.0-2.el9.noarch.rpm 1.4 MB/s | 36 kB 00:00 2024-09-22T03:05:39.072 INFO:teuthology.orchestra.run.smithi196.stdout:(30/38): python3-logutils-0.3.5-21.el9.noarch.r 552 kB/s | 46 kB 00:00 2024-09-22T03:05:39.081 INFO:teuthology.orchestra.run.smithi153.stdout:(34/38): python3-webob-1.8.8-2.el9.noarch.rpm 6.8 MB/s | 230 kB 00:00 2024-09-22T03:05:39.097 INFO:teuthology.orchestra.run.smithi196.stdout:(31/38): python3-portend-3.1.0-2.el9.noarch.rpm 717 kB/s | 16 kB 00:00 2024-09-22T03:05:39.115 INFO:teuthology.orchestra.run.smithi153.stdout:(35/38): python3-werkzeug-2.0.3-3.el9.1.noarch. 12 MB/s | 427 kB 00:00 2024-09-22T03:05:39.122 INFO:teuthology.orchestra.run.smithi196.stdout:(32/38): python3-more-itertools-8.12.0-2.el9.no 723 kB/s | 79 kB 00:00 2024-09-22T03:05:39.140 INFO:teuthology.orchestra.run.smithi153.stdout:(36/38): python3-zc-lockfile-2.0-10.el9.noarch. 808 kB/s | 20 kB 00:00 2024-09-22T03:05:39.147 INFO:teuthology.orchestra.run.smithi196.stdout:(33/38): python3-pecan-1.4.2-3.el9.noarch.rpm 2.6 MB/s | 272 kB 00:00 2024-09-22T03:05:39.166 INFO:teuthology.orchestra.run.smithi153.stdout:(37/38): lua-devel-5.4.4-4.el9.x86_64.rpm 38 kB/s | 22 kB 00:00 2024-09-22T03:05:39.172 INFO:teuthology.orchestra.run.smithi196.stdout:(34/38): python3-tempora-5.0.0-2.el9.noarch.rpm 714 kB/s | 36 kB 00:00 2024-09-22T03:05:39.198 INFO:teuthology.orchestra.run.smithi196.stdout:(35/38): python3-pyOpenSSL-21.0.0-1.el9.noarch. 892 kB/s | 90 kB 00:00 2024-09-22T03:05:39.223 INFO:teuthology.orchestra.run.smithi196.stdout:(36/38): python3-zc-lockfile-2.0-10.el9.noarch. 806 kB/s | 20 kB 00:00 2024-09-22T03:05:39.248 INFO:teuthology.orchestra.run.smithi196.stdout:(37/38): python3-webob-1.8.8-2.el9.noarch.rpm 2.2 MB/s | 230 kB 00:00 2024-09-22T03:05:39.273 INFO:teuthology.orchestra.run.smithi196.stdout:(38/38): python3-werkzeug-2.0.3-3.el9.1.noarch. 4.2 MB/s | 427 kB 00:00 2024-09-22T03:05:39.278 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-09-22T03:05:39.279 INFO:teuthology.orchestra.run.smithi196.stdout:Total 26 MB/s | 134 MB 00:05 2024-09-22T03:05:39.446 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-09-22T03:05:39.487 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-09-22T03:05:39.487 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-09-22T03:05:40.569 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-09-22T03:05:40.569 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-09-22T03:05:41.616 INFO:teuthology.orchestra.run.smithi153.stdout:(38/38): ceph-osd-19.1.1-373.g4e45e6b4.el9.x86_ 30 MB/s | 120 MB 00:04 2024-09-22T03:05:41.620 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-09-22T03:05:41.620 INFO:teuthology.orchestra.run.smithi153.stdout:Total 27 MB/s | 134 MB 00:05 2024-09-22T03:05:41.790 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-09-22T03:05:41.830 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-09-22T03:05:41.831 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-09-22T03:05:42.636 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-09-22T03:05:42.701 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/38 2024-09-22T03:05:42.739 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/38 2024-09-22T03:05:42.770 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/38 2024-09-22T03:05:42.840 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/38 2024-09-22T03:05:42.886 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : luarocks-3.9.2-5.el9.noarch 5/38 2024-09-22T03:05:42.904 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-09-22T03:05:42.905 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-09-22T03:05:42.919 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/38 2024-09-22T03:05:43.010 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/38 2024-09-22T03:05:43.089 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/38 2024-09-22T03:05:43.138 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 9/38 2024-09-22T03:05:43.173 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/38 2024-09-22T03:05:43.203 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/38 2024-09-22T03:05:43.237 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/38 2024-09-22T03:05:43.270 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/38 2024-09-22T03:05:43.311 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/38 2024-09-22T03:05:43.354 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/38 2024-09-22T03:05:43.389 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/38 2024-09-22T03:05:43.438 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/38 2024-09-22T03:05:43.470 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-packaging-20.9-5.el9.noarch 18/38 2024-09-22T03:05:43.551 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 19/38 2024-09-22T03:05:43.716 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 20/38 2024-09-22T03:05:43.777 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 21/38 2024-09-22T03:05:44.895 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-09-22T03:05:44.952 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/38 2024-09-22T03:05:44.990 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/38 2024-09-22T03:05:45.021 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/38 2024-09-22T03:05:45.094 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/38 2024-09-22T03:05:45.145 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : luarocks-3.9.2-5.el9.noarch 5/38 2024-09-22T03:05:45.177 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/38 2024-09-22T03:05:45.271 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/38 2024-09-22T03:05:45.356 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/38 2024-09-22T03:05:45.405 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 9/38 2024-09-22T03:05:45.440 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/38 2024-09-22T03:05:45.470 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/38 2024-09-22T03:05:45.504 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/38 2024-09-22T03:05:45.537 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/38 2024-09-22T03:05:45.586 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/38 2024-09-22T03:05:45.638 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/38 2024-09-22T03:05:45.672 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/38 2024-09-22T03:05:45.720 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/38 2024-09-22T03:05:45.752 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-packaging-20.9-5.el9.noarch 18/38 2024-09-22T03:05:45.826 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 19/38 2024-09-22T03:05:45.974 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 20/38 2024-09-22T03:05:46.024 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 21/38 2024-09-22T03:06:02.607 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 22/38 2024-09-22T03:06:02.640 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-osd-2:19.1.1-373.g4e45e6b4.el9.x86_64 23/38 2024-09-22T03:06:02.668 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-osd-2:19.1.1-373.g4e45e6b4.el9.x86_64 23/38 2024-09-22T03:06:02.669 INFO:teuthology.orchestra.run.smithi196.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-22T03:06:02.669 INFO:teuthology.orchestra.run.smithi196.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-22T03:06:02.669 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-22T03:06:02.669 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-22T03:06:02.669 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-22T03:06:02.766 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 24/38 2024-09-22T03:06:02.805 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 25/38 2024-09-22T03:06:02.853 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-ply-3.11-14.el9.noarch 26/38 2024-09-22T03:06:02.908 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 27/38 2024-09-22T03:06:03.092 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 28/38 2024-09-22T03:06:03.142 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 29/38 2024-09-22T03:06:03.211 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 30/38 2024-09-22T03:06:03.300 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 31/38 2024-09-22T03:06:03.342 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 32/38 2024-09-22T03:06:03.457 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 33/38 2024-09-22T03:06:03.619 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-mgr-modules-core-2:19.1.1-373.g4e45e6b4.el9.n 34/38 2024-09-22T03:06:03.644 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-mgr-2:19.1.1-373.g4e45e6b4.el9.x86_64 35/38 2024-09-22T03:06:03.669 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-mgr-2:19.1.1-373.g4e45e6b4.el9.x86_64 35/38 2024-09-22T03:06:03.669 INFO:teuthology.orchestra.run.smithi196.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-22T03:06:03.669 INFO:teuthology.orchestra.run.smithi196.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-22T03:06:03.669 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-22T03:06:03.669 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-22T03:06:03.669 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-22T03:06:04.146 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-mon-2:19.1.1-373.g4e45e6b4.el9.x86_64 36/38 2024-09-22T03:06:04.168 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-mon-2:19.1.1-373.g4e45e6b4.el9.x86_64 36/38 2024-09-22T03:06:04.169 INFO:teuthology.orchestra.run.smithi196.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-22T03:06:04.169 INFO:teuthology.orchestra.run.smithi196.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-22T03:06:04.169 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-22T03:06:04.169 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-22T03:06:04.169 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-22T03:06:04.404 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-mds-2:19.1.1-373.g4e45e6b4.el9.x86_64 37/38 2024-09-22T03:06:04.428 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-mds-2:19.1.1-373.g4e45e6b4.el9.x86_64 37/38 2024-09-22T03:06:04.428 INFO:teuthology.orchestra.run.smithi196.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-22T03:06:04.428 INFO:teuthology.orchestra.run.smithi196.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-22T03:06:04.428 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-22T03:06:04.428 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-22T03:06:04.428 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-22T03:06:04.501 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-2:19.1.1-373.g4e45e6b4.el9.x86_64 38/38 2024-09-22T03:06:04.922 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 22/38 2024-09-22T03:06:04.954 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-osd-2:19.1.1-373.g4e45e6b4.el9.x86_64 23/38 2024-09-22T03:06:04.982 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-osd-2:19.1.1-373.g4e45e6b4.el9.x86_64 23/38 2024-09-22T03:06:04.982 INFO:teuthology.orchestra.run.smithi153.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-22T03:06:04.982 INFO:teuthology.orchestra.run.smithi153.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-22T03:06:04.982 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-22T03:06:04.983 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-22T03:06:04.983 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-22T03:06:05.079 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 24/38 2024-09-22T03:06:05.117 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 25/38 2024-09-22T03:06:05.175 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-ply-3.11-14.el9.noarch 26/38 2024-09-22T03:06:05.230 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 27/38 2024-09-22T03:06:05.429 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 28/38 2024-09-22T03:06:05.473 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 29/38 2024-09-22T03:06:05.552 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 30/38 2024-09-22T03:06:05.640 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 31/38 2024-09-22T03:06:05.714 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 32/38 2024-09-22T03:06:05.838 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 33/38 2024-09-22T03:06:06.005 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-mgr-modules-core-2:19.1.1-373.g4e45e6b4.el9.n 34/38 2024-09-22T03:06:06.032 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-mgr-2:19.1.1-373.g4e45e6b4.el9.x86_64 35/38 2024-09-22T03:06:06.056 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-mgr-2:19.1.1-373.g4e45e6b4.el9.x86_64 35/38 2024-09-22T03:06:06.056 INFO:teuthology.orchestra.run.smithi153.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-22T03:06:06.056 INFO:teuthology.orchestra.run.smithi153.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-22T03:06:06.056 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-22T03:06:06.056 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-22T03:06:06.056 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-22T03:06:06.516 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-mon-2:19.1.1-373.g4e45e6b4.el9.x86_64 36/38 2024-09-22T03:06:06.540 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-mon-2:19.1.1-373.g4e45e6b4.el9.x86_64 36/38 2024-09-22T03:06:06.540 INFO:teuthology.orchestra.run.smithi153.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-22T03:06:06.540 INFO:teuthology.orchestra.run.smithi153.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-22T03:06:06.540 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-22T03:06:06.540 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-22T03:06:06.540 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-22T03:06:06.767 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-mds-2:19.1.1-373.g4e45e6b4.el9.x86_64 37/38 2024-09-22T03:06:06.790 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-mds-2:19.1.1-373.g4e45e6b4.el9.x86_64 37/38 2024-09-22T03:06:06.790 INFO:teuthology.orchestra.run.smithi153.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-22T03:06:06.790 INFO:teuthology.orchestra.run.smithi153.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-22T03:06:06.790 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-22T03:06:06.791 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-22T03:06:06.791 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-22T03:06:06.854 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-2:19.1.1-373.g4e45e6b4.el9.x86_64 38/38 2024-09-22T03:06:14.637 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-2:19.1.1-373.g4e45e6b4.el9.x86_64 38/38 2024-09-22T03:06:14.637 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/38 2024-09-22T03:06:14.637 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-mds-2:19.1.1-373.g4e45e6b4.el9.x86_64 2/38 2024-09-22T03:06:14.637 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-mgr-2:19.1.1-373.g4e45e6b4.el9.x86_64 3/38 2024-09-22T03:06:14.637 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-mon-2:19.1.1-373.g4e45e6b4.el9.x86_64 4/38 2024-09-22T03:06:14.638 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-osd-2:19.1.1-373.g4e45e6b4.el9.x86_64 5/38 2024-09-22T03:06:14.638 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-mgr-modules-core-2:19.1.1-373.g4e45e6b4.el9.n 6/38 2024-09-22T03:06:14.638 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/38 2024-09-22T03:06:14.638 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/38 2024-09-22T03:06:14.638 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/38 2024-09-22T03:06:14.638 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/38 2024-09-22T03:06:14.638 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/38 2024-09-22T03:06:14.638 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/38 2024-09-22T03:06:14.638 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/38 2024-09-22T03:06:14.638 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 14/38 2024-09-22T03:06:14.638 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 15/38 2024-09-22T03:06:14.638 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 16/38 2024-09-22T03:06:14.638 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 17/38 2024-09-22T03:06:14.639 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 18/38 2024-09-22T03:06:14.639 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 19/38 2024-09-22T03:06:14.639 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 20/38 2024-09-22T03:06:14.639 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 21/38 2024-09-22T03:06:14.639 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 22/38 2024-09-22T03:06:14.639 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 23/38 2024-09-22T03:06:14.639 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 24/38 2024-09-22T03:06:14.639 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 25/38 2024-09-22T03:06:14.639 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 26/38 2024-09-22T03:06:14.639 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 27/38 2024-09-22T03:06:14.639 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 28/38 2024-09-22T03:06:14.640 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 29/38 2024-09-22T03:06:14.640 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 30/38 2024-09-22T03:06:14.640 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 31/38 2024-09-22T03:06:14.641 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 32/38 2024-09-22T03:06:14.641 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 33/38 2024-09-22T03:06:14.641 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 34/38 2024-09-22T03:06:14.641 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 35/38 2024-09-22T03:06:14.641 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 36/38 2024-09-22T03:06:14.641 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 37/38 2024-09-22T03:06:15.564 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 38/38 2024-09-22T03:06:15.565 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-22T03:06:15.565 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-09-22T03:06:15.565 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T03:06:15.565 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mds-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T03:06:15.565 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mgr-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T03:06:15.565 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mgr-modules-core-2:19.1.1-373.g4e45e6b4.el9.noarch 2024-09-22T03:06:15.565 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mon-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T03:06:15.565 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-osd-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T03:06:15.565 INFO:teuthology.orchestra.run.smithi196.stdout: lua-5.4.4-4.el9.x86_64 2024-09-22T03:06:15.565 INFO:teuthology.orchestra.run.smithi196.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-09-22T03:06:15.565 INFO:teuthology.orchestra.run.smithi196.stdout: luarocks-3.9.2-5.el9.noarch 2024-09-22T03:06:15.565 INFO:teuthology.orchestra.run.smithi196.stdout: protobuf-3.14.0-13.el9.x86_64 2024-09-22T03:06:15.566 INFO:teuthology.orchestra.run.smithi196.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-22T03:06:15.566 INFO:teuthology.orchestra.run.smithi196.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-22T03:06:15.566 INFO:teuthology.orchestra.run.smithi196.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-22T03:06:15.566 INFO:teuthology.orchestra.run.smithi196.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-22T03:06:15.566 INFO:teuthology.orchestra.run.smithi196.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-22T03:06:15.566 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-22T03:06:15.566 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-22T03:06:15.566 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-22T03:06:15.566 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-22T03:06:15.566 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-22T03:06:15.566 INFO:teuthology.orchestra.run.smithi196.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-22T03:06:15.566 INFO:teuthology.orchestra.run.smithi196.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-22T03:06:15.566 INFO:teuthology.orchestra.run.smithi196.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-22T03:06:15.566 INFO:teuthology.orchestra.run.smithi196.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-22T03:06:15.567 INFO:teuthology.orchestra.run.smithi196.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-22T03:06:15.567 INFO:teuthology.orchestra.run.smithi196.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-22T03:06:15.567 INFO:teuthology.orchestra.run.smithi196.stdout: python3-ply-3.11-14.el9.noarch 2024-09-22T03:06:15.567 INFO:teuthology.orchestra.run.smithi196.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-22T03:06:15.567 INFO:teuthology.orchestra.run.smithi196.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-22T03:06:15.567 INFO:teuthology.orchestra.run.smithi196.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-22T03:06:15.567 INFO:teuthology.orchestra.run.smithi196.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-22T03:06:15.567 INFO:teuthology.orchestra.run.smithi196.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-22T03:06:15.567 INFO:teuthology.orchestra.run.smithi196.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-22T03:06:15.567 INFO:teuthology.orchestra.run.smithi196.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-22T03:06:15.567 INFO:teuthology.orchestra.run.smithi196.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-22T03:06:15.567 INFO:teuthology.orchestra.run.smithi196.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-22T03:06:15.567 INFO:teuthology.orchestra.run.smithi196.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-22T03:06:15.568 INFO:teuthology.orchestra.run.smithi196.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-22T03:06:15.568 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-22T03:06:15.568 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-09-22T03:06:15.901 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install ceph-base 2024-09-22T03:06:16.474 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:01:45 ago on Sun 22 Sep 2024 03:04:31 AM UTC. 2024-09-22T03:06:16.562 INFO:teuthology.orchestra.run.smithi196.stdout:Package ceph-base-2:19.1.1-373.g4e45e6b4.el9.x86_64 is already installed. 2024-09-22T03:06:16.605 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-09-22T03:06:16.606 INFO:teuthology.orchestra.run.smithi196.stdout:Nothing to do. 2024-09-22T03:06:16.606 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-09-22T03:06:16.674 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install cephadm 2024-09-22T03:06:16.988 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-2:19.1.1-373.g4e45e6b4.el9.x86_64 38/38 2024-09-22T03:06:16.988 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/38 2024-09-22T03:06:16.988 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-mds-2:19.1.1-373.g4e45e6b4.el9.x86_64 2/38 2024-09-22T03:06:16.988 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-mgr-2:19.1.1-373.g4e45e6b4.el9.x86_64 3/38 2024-09-22T03:06:16.988 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-mon-2:19.1.1-373.g4e45e6b4.el9.x86_64 4/38 2024-09-22T03:06:16.989 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-osd-2:19.1.1-373.g4e45e6b4.el9.x86_64 5/38 2024-09-22T03:06:16.989 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-mgr-modules-core-2:19.1.1-373.g4e45e6b4.el9.n 6/38 2024-09-22T03:06:16.989 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/38 2024-09-22T03:06:16.989 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/38 2024-09-22T03:06:16.989 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/38 2024-09-22T03:06:16.989 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/38 2024-09-22T03:06:16.989 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/38 2024-09-22T03:06:16.989 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/38 2024-09-22T03:06:16.989 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/38 2024-09-22T03:06:16.989 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 14/38 2024-09-22T03:06:16.989 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 15/38 2024-09-22T03:06:16.989 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 16/38 2024-09-22T03:06:16.989 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 17/38 2024-09-22T03:06:16.989 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 18/38 2024-09-22T03:06:16.990 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 19/38 2024-09-22T03:06:16.990 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 20/38 2024-09-22T03:06:16.990 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 21/38 2024-09-22T03:06:16.990 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 22/38 2024-09-22T03:06:16.990 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 23/38 2024-09-22T03:06:16.990 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 24/38 2024-09-22T03:06:16.990 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 25/38 2024-09-22T03:06:16.990 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 26/38 2024-09-22T03:06:16.990 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 27/38 2024-09-22T03:06:16.990 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 28/38 2024-09-22T03:06:16.990 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 29/38 2024-09-22T03:06:16.990 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 30/38 2024-09-22T03:06:16.990 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 31/38 2024-09-22T03:06:16.991 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 32/38 2024-09-22T03:06:16.991 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 33/38 2024-09-22T03:06:16.991 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 34/38 2024-09-22T03:06:16.991 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 35/38 2024-09-22T03:06:16.991 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 36/38 2024-09-22T03:06:16.991 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 37/38 2024-09-22T03:06:17.255 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:01:46 ago on Sun 22 Sep 2024 03:04:31 AM UTC. 2024-09-22T03:06:17.386 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-09-22T03:06:17.387 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-09-22T03:06:17.387 INFO:teuthology.orchestra.run.smithi196.stdout: Package Arch Version Repository Size 2024-09-22T03:06:17.387 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-09-22T03:06:17.387 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-09-22T03:06:17.387 INFO:teuthology.orchestra.run.smithi196.stdout: cephadm noarch 2:19.1.1-373.g4e45e6b4.el9 ceph-noarch 765 k 2024-09-22T03:06:17.387 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-22T03:06:17.388 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-09-22T03:06:17.388 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-09-22T03:06:17.388 INFO:teuthology.orchestra.run.smithi196.stdout:Install 1 Package 2024-09-22T03:06:17.388 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-22T03:06:17.388 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 765 k 2024-09-22T03:06:17.388 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 771 k 2024-09-22T03:06:17.388 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-09-22T03:06:17.677 INFO:teuthology.orchestra.run.smithi196.stdout:cephadm-19.1.1-373.g4e45e6b4.el9.noarch.rpm 2.6 MB/s | 765 kB 00:00 2024-09-22T03:06:17.677 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-09-22T03:06:17.678 INFO:teuthology.orchestra.run.smithi196.stdout:Total 2.6 MB/s | 765 kB 00:00 2024-09-22T03:06:17.678 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-09-22T03:06:17.683 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-09-22T03:06:17.683 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-09-22T03:06:17.689 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-09-22T03:06:17.689 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-09-22T03:06:17.766 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-09-22T03:06:18.137 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 38/38 2024-09-22T03:06:18.137 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-22T03:06:18.137 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-09-22T03:06:18.137 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T03:06:18.137 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mds-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T03:06:18.137 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T03:06:18.137 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-modules-core-2:19.1.1-373.g4e45e6b4.el9.noarch 2024-09-22T03:06:18.137 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mon-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T03:06:18.137 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-osd-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T03:06:18.137 INFO:teuthology.orchestra.run.smithi153.stdout: lua-5.4.4-4.el9.x86_64 2024-09-22T03:06:18.137 INFO:teuthology.orchestra.run.smithi153.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-09-22T03:06:18.137 INFO:teuthology.orchestra.run.smithi153.stdout: luarocks-3.9.2-5.el9.noarch 2024-09-22T03:06:18.137 INFO:teuthology.orchestra.run.smithi153.stdout: protobuf-3.14.0-13.el9.x86_64 2024-09-22T03:06:18.138 INFO:teuthology.orchestra.run.smithi153.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-22T03:06:18.138 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-22T03:06:18.138 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-22T03:06:18.138 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-22T03:06:18.138 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-22T03:06:18.138 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-22T03:06:18.138 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-22T03:06:18.138 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-22T03:06:18.138 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-22T03:06:18.138 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-22T03:06:18.138 INFO:teuthology.orchestra.run.smithi153.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-22T03:06:18.139 INFO:teuthology.orchestra.run.smithi153.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-22T03:06:18.139 INFO:teuthology.orchestra.run.smithi153.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-22T03:06:18.139 INFO:teuthology.orchestra.run.smithi153.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-22T03:06:18.139 INFO:teuthology.orchestra.run.smithi153.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-22T03:06:18.139 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-22T03:06:18.139 INFO:teuthology.orchestra.run.smithi153.stdout: python3-ply-3.11-14.el9.noarch 2024-09-22T03:06:18.139 INFO:teuthology.orchestra.run.smithi153.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-22T03:06:18.139 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-22T03:06:18.139 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-22T03:06:18.139 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-22T03:06:18.139 INFO:teuthology.orchestra.run.smithi153.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-22T03:06:18.139 INFO:teuthology.orchestra.run.smithi153.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-22T03:06:18.139 INFO:teuthology.orchestra.run.smithi153.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-22T03:06:18.139 INFO:teuthology.orchestra.run.smithi153.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-22T03:06:18.139 INFO:teuthology.orchestra.run.smithi153.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-22T03:06:18.140 INFO:teuthology.orchestra.run.smithi153.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-22T03:06:18.140 INFO:teuthology.orchestra.run.smithi153.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-22T03:06:18.140 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-22T03:06:18.140 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-09-22T03:06:18.221 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: cephadm-2:19.1.1-373.g4e45e6b4.el9.noarch 1/1 2024-09-22T03:06:18.315 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : cephadm-2:19.1.1-373.g4e45e6b4.el9.noarch 1/1 2024-09-22T03:06:18.405 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install ceph-base 2024-09-22T03:06:18.649 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: cephadm-2:19.1.1-373.g4e45e6b4.el9.noarch 1/1 2024-09-22T03:06:18.906 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : cephadm-2:19.1.1-373.g4e45e6b4.el9.noarch 1/1 2024-09-22T03:06:18.907 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-22T03:06:18.907 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-09-22T03:06:18.907 INFO:teuthology.orchestra.run.smithi196.stdout: cephadm-2:19.1.1-373.g4e45e6b4.el9.noarch 2024-09-22T03:06:18.907 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-22T03:06:18.907 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-09-22T03:06:18.983 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:46 ago on Sun 22 Sep 2024 03:04:32 AM UTC. 2024-09-22T03:06:19.071 INFO:teuthology.orchestra.run.smithi153.stdout:Package ceph-base-2:19.1.1-373.g4e45e6b4.el9.x86_64 is already installed. 2024-09-22T03:06:19.113 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-09-22T03:06:19.115 INFO:teuthology.orchestra.run.smithi153.stdout:Nothing to do. 2024-09-22T03:06:19.115 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-09-22T03:06:19.156 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install ceph-immutable-object-cache 2024-09-22T03:06:19.192 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install cephadm 2024-09-22T03:06:19.746 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:01:48 ago on Sun 22 Sep 2024 03:04:31 AM UTC. 2024-09-22T03:06:19.766 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:47 ago on Sun 22 Sep 2024 03:04:32 AM UTC. 2024-09-22T03:06:19.883 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-09-22T03:06:19.884 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-09-22T03:06:19.884 INFO:teuthology.orchestra.run.smithi196.stdout: Package Arch Version Repo Size 2024-09-22T03:06:19.884 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-09-22T03:06:19.884 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-09-22T03:06:19.884 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-immutable-object-cache x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 147 k 2024-09-22T03:06:19.884 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-22T03:06:19.884 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-09-22T03:06:19.884 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-09-22T03:06:19.884 INFO:teuthology.orchestra.run.smithi196.stdout:Install 1 Package 2024-09-22T03:06:19.884 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-22T03:06:19.885 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 147 k 2024-09-22T03:06:19.885 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 481 k 2024-09-22T03:06:19.885 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-09-22T03:06:19.897 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-09-22T03:06:19.898 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-09-22T03:06:19.899 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repository Size 2024-09-22T03:06:19.899 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-09-22T03:06:19.899 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-09-22T03:06:19.899 INFO:teuthology.orchestra.run.smithi153.stdout: cephadm noarch 2:19.1.1-373.g4e45e6b4.el9 ceph-noarch 765 k 2024-09-22T03:06:19.899 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-22T03:06:19.899 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-09-22T03:06:19.899 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-09-22T03:06:19.899 INFO:teuthology.orchestra.run.smithi153.stdout:Install 1 Package 2024-09-22T03:06:19.899 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-22T03:06:19.899 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 765 k 2024-09-22T03:06:19.899 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 771 k 2024-09-22T03:06:19.899 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-09-22T03:06:20.099 INFO:teuthology.orchestra.run.smithi196.stdout:ceph-immutable-object-cache-19.1.1-373.g4e45e6b 688 kB/s | 147 kB 00:00 2024-09-22T03:06:20.100 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-09-22T03:06:20.100 INFO:teuthology.orchestra.run.smithi196.stdout:Total 683 kB/s | 147 kB 00:00 2024-09-22T03:06:20.100 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-09-22T03:06:20.107 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-09-22T03:06:20.107 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-09-22T03:06:20.156 INFO:teuthology.orchestra.run.smithi153.stdout:cephadm-19.1.1-373.g4e45e6b4.el9.noarch.rpm 2.9 MB/s | 765 kB 00:00 2024-09-22T03:06:20.156 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-09-22T03:06:20.157 INFO:teuthology.orchestra.run.smithi153.stdout:Total 2.9 MB/s | 765 kB 00:00 2024-09-22T03:06:20.157 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-09-22T03:06:20.162 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-09-22T03:06:20.162 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-09-22T03:06:20.164 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-09-22T03:06:20.165 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-09-22T03:06:20.168 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-09-22T03:06:20.168 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-09-22T03:06:20.241 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-09-22T03:06:20.310 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-09-22T03:06:20.375 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-immutable-object-cache-2:19.1.1-373.g4e45e6b4.e 1/1 2024-09-22T03:06:20.400 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.1.1-373.g4e45e6b4.e 1/1 2024-09-22T03:06:20.400 INFO:teuthology.orchestra.run.smithi196.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-22T03:06:20.400 INFO:teuthology.orchestra.run.smithi196.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-22T03:06:20.400 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-22T03:06:20.674 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: cephadm-2:19.1.1-373.g4e45e6b4.el9.noarch 1/1 2024-09-22T03:06:20.768 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : cephadm-2:19.1.1-373.g4e45e6b4.el9.noarch 1/1 2024-09-22T03:06:21.162 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: cephadm-2:19.1.1-373.g4e45e6b4.el9.noarch 1/1 2024-09-22T03:06:21.332 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-immutable-object-cache-2:19.1.1-373.g4e45e6b4.e 1/1 2024-09-22T03:06:21.332 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-22T03:06:21.332 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-09-22T03:06:21.332 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-immutable-object-cache-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T03:06:21.332 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-22T03:06:21.332 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-09-22T03:06:21.414 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : cephadm-2:19.1.1-373.g4e45e6b4.el9.noarch 1/1 2024-09-22T03:06:21.414 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-22T03:06:21.414 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-09-22T03:06:21.414 INFO:teuthology.orchestra.run.smithi153.stdout: cephadm-2:19.1.1-373.g4e45e6b4.el9.noarch 2024-09-22T03:06:21.414 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-22T03:06:21.414 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-09-22T03:06:21.570 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install ceph-mgr 2024-09-22T03:06:21.618 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install ceph-immutable-object-cache 2024-09-22T03:06:22.159 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:01:51 ago on Sun 22 Sep 2024 03:04:31 AM UTC. 2024-09-22T03:06:22.211 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:50 ago on Sun 22 Sep 2024 03:04:32 AM UTC. 2024-09-22T03:06:22.254 INFO:teuthology.orchestra.run.smithi196.stdout:Package ceph-mgr-2:19.1.1-373.g4e45e6b4.el9.x86_64 is already installed. 2024-09-22T03:06:22.297 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-09-22T03:06:22.298 INFO:teuthology.orchestra.run.smithi196.stdout:Nothing to do. 2024-09-22T03:06:22.298 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-09-22T03:06:22.344 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-09-22T03:06:22.344 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-09-22T03:06:22.344 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repo Size 2024-09-22T03:06:22.345 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-09-22T03:06:22.345 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-09-22T03:06:22.345 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-immutable-object-cache x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 147 k 2024-09-22T03:06:22.345 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-22T03:06:22.345 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-09-22T03:06:22.345 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-09-22T03:06:22.345 INFO:teuthology.orchestra.run.smithi153.stdout:Install 1 Package 2024-09-22T03:06:22.345 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-22T03:06:22.346 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 147 k 2024-09-22T03:06:22.346 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 481 k 2024-09-22T03:06:22.346 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-09-22T03:06:22.375 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install ceph-mgr-dashboard 2024-09-22T03:06:22.528 INFO:teuthology.orchestra.run.smithi153.stdout:ceph-immutable-object-cache-19.1.1-373.g4e45e6b 810 kB/s | 147 kB 00:00 2024-09-22T03:06:22.529 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-09-22T03:06:22.529 INFO:teuthology.orchestra.run.smithi153.stdout:Total 802 kB/s | 147 kB 00:00 2024-09-22T03:06:22.529 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-09-22T03:06:22.537 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-09-22T03:06:22.537 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-09-22T03:06:22.594 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-09-22T03:06:22.594 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-09-22T03:06:22.736 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-09-22T03:06:22.821 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-immutable-object-cache-2:19.1.1-373.g4e45e6b4.e 1/1 2024-09-22T03:06:22.845 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.1.1-373.g4e45e6b4.e 1/1 2024-09-22T03:06:22.845 INFO:teuthology.orchestra.run.smithi153.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-22T03:06:22.845 INFO:teuthology.orchestra.run.smithi153.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-22T03:06:22.846 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-22T03:06:22.950 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:01:51 ago on Sun 22 Sep 2024 03:04:31 AM UTC. 2024-09-22T03:06:23.081 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-09-22T03:06:23.082 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-09-22T03:06:23.082 INFO:teuthology.orchestra.run.smithi196.stdout: Package Arch Version Repository Size 2024-09-22T03:06:23.082 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-09-22T03:06:23.082 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-09-22T03:06:23.082 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mgr-dashboard noarch 2:19.1.1-373.g4e45e6b4.el9 ceph-noarch 3.6 M 2024-09-22T03:06:23.083 INFO:teuthology.orchestra.run.smithi196.stdout:Installing dependencies: 2024-09-22T03:06:23.083 INFO:teuthology.orchestra.run.smithi196.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-09-22T03:06:23.083 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-grafana-dashboards noarch 2:19.1.1-373.g4e45e6b4.el9 ceph-noarch 32 k 2024-09-22T03:06:23.083 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-prometheus-alerts noarch 2:19.1.1-373.g4e45e6b4.el9 ceph-noarch 16 k 2024-09-22T03:06:23.083 INFO:teuthology.orchestra.run.smithi196.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-09-22T03:06:23.083 INFO:teuthology.orchestra.run.smithi196.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-09-22T03:06:23.083 INFO:teuthology.orchestra.run.smithi196.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-09-22T03:06:23.083 INFO:teuthology.orchestra.run.smithi196.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-09-22T03:06:23.083 INFO:teuthology.orchestra.run.smithi196.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-09-22T03:06:23.083 INFO:teuthology.orchestra.run.smithi196.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-22T03:06:23.083 INFO:teuthology.orchestra.run.smithi196.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-22T03:06:23.083 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-22T03:06:23.084 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-09-22T03:06:23.084 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-09-22T03:06:23.084 INFO:teuthology.orchestra.run.smithi196.stdout:Install 11 Packages 2024-09-22T03:06:23.084 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-22T03:06:23.084 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 7.7 M 2024-09-22T03:06:23.084 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 101 M 2024-09-22T03:06:23.085 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-09-22T03:06:23.788 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-immutable-object-cache-2:19.1.1-373.g4e45e6b4.e 1/1 2024-09-22T03:06:23.788 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-22T03:06:23.788 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-09-22T03:06:23.788 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-immutable-object-cache-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T03:06:23.788 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-22T03:06:23.789 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-09-22T03:06:23.845 INFO:teuthology.orchestra.run.smithi196.stdout:(1/11): ceph-grafana-dashboards-19.1.1-373.g4e4 207 kB/s | 32 kB 00:00 2024-09-22T03:06:23.870 INFO:teuthology.orchestra.run.smithi196.stdout:(2/11): ceph-prometheus-alerts-19.1.1-373.g4e45 92 kB/s | 16 kB 00:00 2024-09-22T03:06:24.023 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install ceph-mgr 2024-09-22T03:06:24.095 INFO:teuthology.orchestra.run.smithi196.stdout:(3/11): ceph-mgr-dashboard-19.1.1-373.g4e45e6b4 8.8 MB/s | 3.6 MB 00:00 2024-09-22T03:06:24.145 INFO:teuthology.orchestra.run.smithi196.stdout:(4/11): abseil-cpp-20211102.0-3.el9.x86_64.rpm 11 MB/s | 548 kB 00:00 2024-09-22T03:06:24.170 INFO:teuthology.orchestra.run.smithi196.stdout:(5/11): grpc-data-1.46.7-10.el9.noarch.rpm 784 kB/s | 19 kB 00:00 2024-09-22T03:06:24.246 INFO:teuthology.orchestra.run.smithi196.stdout:(6/11): python3-grpcio-1.46.7-10.el9.x86_64.rpm 27 MB/s | 2.0 MB 00:00 2024-09-22T03:06:24.279 INFO:teuthology.orchestra.run.smithi196.stdout:(7/11): protobuf-compiler-3.14.0-13.el9.x86_64. 2.1 MB/s | 863 kB 00:00 2024-09-22T03:06:24.313 INFO:teuthology.orchestra.run.smithi196.stdout:(8/11): python3-grpcio-tools-1.46.7-10.el9.x86_ 2.1 MB/s | 144 kB 00:00 2024-09-22T03:06:24.338 INFO:teuthology.orchestra.run.smithi196.stdout:(9/11): python3-routes-2.5.1-5.el9.noarch.rpm 7.9 MB/s | 188 kB 00:00 2024-09-22T03:06:24.355 INFO:teuthology.orchestra.run.smithi196.stdout:(10/11): python3-repoze-lru-0.7-16.el9.noarch.r 409 kB/s | 31 kB 00:00 2024-09-22T03:06:24.505 INFO:teuthology.orchestra.run.smithi196.stdout:(11/11): python3-protobuf-3.14.0-13.el9.noarch. 408 kB/s | 269 kB 00:00 2024-09-22T03:06:24.510 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-09-22T03:06:24.510 INFO:teuthology.orchestra.run.smithi196.stdout:Total 5.4 MB/s | 7.7 MB 00:01 2024-09-22T03:06:24.543 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-09-22T03:06:24.561 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-09-22T03:06:24.562 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-09-22T03:06:24.613 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:52 ago on Sun 22 Sep 2024 03:04:32 AM UTC. 2024-09-22T03:06:24.703 INFO:teuthology.orchestra.run.smithi153.stdout:Package ceph-mgr-2:19.1.1-373.g4e45e6b4.el9.x86_64 is already installed. 2024-09-22T03:06:24.745 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-09-22T03:06:24.746 INFO:teuthology.orchestra.run.smithi153.stdout:Nothing to do. 2024-09-22T03:06:24.746 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-09-22T03:06:24.765 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-09-22T03:06:24.766 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-09-22T03:06:24.816 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install ceph-mgr-dashboard 2024-09-22T03:06:25.254 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-09-22T03:06:25.345 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 1/11 2024-09-22T03:06:25.375 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 2/11 2024-09-22T03:06:25.390 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:53 ago on Sun 22 Sep 2024 03:04:32 AM UTC. 2024-09-22T03:06:25.484 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 3/11 2024-09-22T03:06:25.521 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-09-22T03:06:25.522 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-09-22T03:06:25.523 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repository Size 2024-09-22T03:06:25.523 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-09-22T03:06:25.523 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-09-22T03:06:25.523 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-dashboard noarch 2:19.1.1-373.g4e45e6b4.el9 ceph-noarch 3.6 M 2024-09-22T03:06:25.523 INFO:teuthology.orchestra.run.smithi153.stdout:Installing dependencies: 2024-09-22T03:06:25.523 INFO:teuthology.orchestra.run.smithi153.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-09-22T03:06:25.523 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-grafana-dashboards noarch 2:19.1.1-373.g4e45e6b4.el9 ceph-noarch 32 k 2024-09-22T03:06:25.523 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-prometheus-alerts noarch 2:19.1.1-373.g4e45e6b4.el9 ceph-noarch 16 k 2024-09-22T03:06:25.523 INFO:teuthology.orchestra.run.smithi153.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-09-22T03:06:25.523 INFO:teuthology.orchestra.run.smithi153.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-09-22T03:06:25.523 INFO:teuthology.orchestra.run.smithi153.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-09-22T03:06:25.523 INFO:teuthology.orchestra.run.smithi153.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-09-22T03:06:25.524 INFO:teuthology.orchestra.run.smithi153.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-09-22T03:06:25.524 INFO:teuthology.orchestra.run.smithi153.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-22T03:06:25.524 INFO:teuthology.orchestra.run.smithi153.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-22T03:06:25.524 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-22T03:06:25.524 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-09-22T03:06:25.524 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-09-22T03:06:25.524 INFO:teuthology.orchestra.run.smithi153.stdout:Install 11 Packages 2024-09-22T03:06:25.524 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-22T03:06:25.525 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 7.7 M 2024-09-22T03:06:25.525 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 101 M 2024-09-22T03:06:25.525 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-09-22T03:06:25.618 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 4/11 2024-09-22T03:06:25.679 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 5/11 2024-09-22T03:06:25.762 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 6/11 2024-09-22T03:06:25.822 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 7/11 2024-09-22T03:06:25.851 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 8/11 2024-09-22T03:06:25.883 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-prometheus-alerts-2:19.1.1-373.g4e45e6b4.el9. 9/11 2024-09-22T03:06:26.073 INFO:teuthology.orchestra.run.smithi153.stdout:(1/11): ceph-grafana-dashboards-19.1.1-373.g4e4 229 kB/s | 32 kB 00:00 2024-09-22T03:06:26.099 INFO:teuthology.orchestra.run.smithi153.stdout:(2/11): ceph-prometheus-alerts-19.1.1-373.g4e45 100 kB/s | 16 kB 00:00 2024-09-22T03:06:26.232 INFO:teuthology.orchestra.run.smithi153.stdout:(3/11): python3-protobuf-3.14.0-13.el9.noarch.r 1.7 MB/s | 269 kB 00:00 2024-09-22T03:06:26.291 INFO:teuthology.orchestra.run.smithi153.stdout:(4/11): abseil-cpp-20211102.0-3.el9.x86_64.rpm 9.2 MB/s | 548 kB 00:00 2024-09-22T03:06:26.316 INFO:teuthology.orchestra.run.smithi153.stdout:(5/11): grpc-data-1.46.7-10.el9.noarch.rpm 779 kB/s | 19 kB 00:00 2024-09-22T03:06:26.375 INFO:teuthology.orchestra.run.smithi153.stdout:(6/11): ceph-mgr-dashboard-19.1.1-373.g4e45e6b4 8.1 MB/s | 3.6 MB 00:00 2024-09-22T03:06:26.433 INFO:teuthology.orchestra.run.smithi153.stdout:(7/11): python3-grpcio-tools-1.46.7-10.el9.x86_ 2.4 MB/s | 144 kB 00:00 2024-09-22T03:06:26.467 INFO:teuthology.orchestra.run.smithi153.stdout:(8/11): protobuf-compiler-3.14.0-13.el9.x86_64. 2.3 MB/s | 863 kB 00:00 2024-09-22T03:06:26.484 INFO:teuthology.orchestra.run.smithi153.stdout:(9/11): python3-repoze-lru-0.7-16.el9.noarch.rp 611 kB/s | 31 kB 00:00 2024-09-22T03:06:26.534 INFO:teuthology.orchestra.run.smithi153.stdout:(10/11): python3-grpcio-1.46.7-10.el9.x86_64.rp 9.4 MB/s | 2.0 MB 00:00 2024-09-22T03:06:26.567 INFO:teuthology.orchestra.run.smithi153.stdout:(11/11): python3-routes-2.5.1-5.el9.noarch.rpm 1.8 MB/s | 188 kB 00:00 2024-09-22T03:06:26.571 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-09-22T03:06:26.572 INFO:teuthology.orchestra.run.smithi153.stdout:Total 7.3 MB/s | 7.7 MB 00:01 2024-09-22T03:06:26.601 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-09-22T03:06:26.618 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-09-22T03:06:26.618 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-09-22T03:06:26.835 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-09-22T03:06:26.835 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-09-22T03:06:26.936 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-grafana-dashboards-2:19.1.1-373.g4e45e6b4.el9 10/11 2024-09-22T03:06:26.967 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-mgr-dashboard-2:19.1.1-373.g4e45e6b4.el9.noar 11/11 2024-09-22T03:06:27.320 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-09-22T03:06:27.407 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 1/11 2024-09-22T03:06:27.446 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 2/11 2024-09-22T03:06:27.566 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 3/11 2024-09-22T03:06:27.689 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 4/11 2024-09-22T03:06:27.758 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 5/11 2024-09-22T03:06:27.842 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 6/11 2024-09-22T03:06:27.884 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 7/11 2024-09-22T03:06:27.913 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 8/11 2024-09-22T03:06:27.945 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-prometheus-alerts-2:19.1.1-373.g4e45e6b4.el9. 9/11 2024-09-22T03:06:28.094 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.1.1-373.g4e45e6b4.el9.noar 11/11 2024-09-22T03:06:28.095 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-grafana-dashboards-2:19.1.1-373.g4e45e6b4.el9 1/11 2024-09-22T03:06:28.095 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-mgr-dashboard-2:19.1.1-373.g4e45e6b4.el9.noar 2/11 2024-09-22T03:06:28.095 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-prometheus-alerts-2:19.1.1-373.g4e45e6b4.el9. 3/11 2024-09-22T03:06:28.095 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 4/11 2024-09-22T03:06:28.095 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 5/11 2024-09-22T03:06:28.095 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 6/11 2024-09-22T03:06:28.095 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 7/11 2024-09-22T03:06:28.095 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 8/11 2024-09-22T03:06:28.095 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/11 2024-09-22T03:06:28.095 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 10/11 2024-09-22T03:06:28.413 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 11/11 2024-09-22T03:06:28.413 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-22T03:06:28.414 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-09-22T03:06:28.414 INFO:teuthology.orchestra.run.smithi196.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-09-22T03:06:28.414 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-grafana-dashboards-2:19.1.1-373.g4e45e6b4.el9.noarch 2024-09-22T03:06:28.414 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mgr-dashboard-2:19.1.1-373.g4e45e6b4.el9.noarch 2024-09-22T03:06:28.414 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-prometheus-alerts-2:19.1.1-373.g4e45e6b4.el9.noarch 2024-09-22T03:06:28.415 INFO:teuthology.orchestra.run.smithi196.stdout: grpc-data-1.46.7-10.el9.noarch 2024-09-22T03:06:28.415 INFO:teuthology.orchestra.run.smithi196.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-09-22T03:06:28.415 INFO:teuthology.orchestra.run.smithi196.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-09-22T03:06:28.415 INFO:teuthology.orchestra.run.smithi196.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-09-22T03:06:28.415 INFO:teuthology.orchestra.run.smithi196.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-09-22T03:06:28.415 INFO:teuthology.orchestra.run.smithi196.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-22T03:06:28.416 INFO:teuthology.orchestra.run.smithi196.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-22T03:06:28.416 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-22T03:06:28.416 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-09-22T03:06:28.683 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-22T03:06:29.003 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-grafana-dashboards-2:19.1.1-373.g4e45e6b4.el9 10/11 2024-09-22T03:06:29.045 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-mgr-dashboard-2:19.1.1-373.g4e45e6b4.el9.noar 11/11 2024-09-22T03:06:29.264 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:01:58 ago on Sun 22 Sep 2024 03:04:31 AM UTC. 2024-09-22T03:06:29.399 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-09-22T03:06:29.400 INFO:teuthology.orchestra.run.smithi196.stdout:====================================================================================== 2024-09-22T03:06:29.400 INFO:teuthology.orchestra.run.smithi196.stdout: Package Arch Version Repository Size 2024-09-22T03:06:29.400 INFO:teuthology.orchestra.run.smithi196.stdout:====================================================================================== 2024-09-22T03:06:29.400 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-09-22T03:06:29.400 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mgr-diskprediction-local noarch 2:19.1.1-373.g4e45e6b4.el9 ceph-noarch 7.4 M 2024-09-22T03:06:29.400 INFO:teuthology.orchestra.run.smithi196.stdout:Installing dependencies: 2024-09-22T03:06:29.400 INFO:teuthology.orchestra.run.smithi196.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-22T03:06:29.400 INFO:teuthology.orchestra.run.smithi196.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-22T03:06:29.401 INFO:teuthology.orchestra.run.smithi196.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-22T03:06:29.401 INFO:teuthology.orchestra.run.smithi196.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-22T03:06:29.401 INFO:teuthology.orchestra.run.smithi196.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-22T03:06:29.401 INFO:teuthology.orchestra.run.smithi196.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-22T03:06:29.401 INFO:teuthology.orchestra.run.smithi196.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-22T03:06:29.401 INFO:teuthology.orchestra.run.smithi196.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-22T03:06:29.401 INFO:teuthology.orchestra.run.smithi196.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-22T03:06:29.401 INFO:teuthology.orchestra.run.smithi196.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-22T03:06:29.401 INFO:teuthology.orchestra.run.smithi196.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-22T03:06:29.401 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-22T03:06:29.401 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-09-22T03:06:29.401 INFO:teuthology.orchestra.run.smithi196.stdout:====================================================================================== 2024-09-22T03:06:29.401 INFO:teuthology.orchestra.run.smithi196.stdout:Install 12 Packages 2024-09-22T03:06:29.401 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-22T03:06:29.402 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 38 M 2024-09-22T03:06:29.402 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 212 M 2024-09-22T03:06:29.402 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-09-22T03:06:29.646 INFO:teuthology.orchestra.run.smithi196.stdout:(1/12): libquadmath-11.5.0-2.el9.x86_64.rpm 1.1 MB/s | 191 kB 00:00 2024-09-22T03:06:29.688 INFO:teuthology.orchestra.run.smithi196.stdout:(2/12): libgfortran-11.5.0-2.el9.x86_64.rpm 3.6 MB/s | 801 kB 00:00 2024-09-22T03:06:29.780 INFO:teuthology.orchestra.run.smithi196.stdout:(3/12): flexiblas-3.0.4-8.el9.x86_64.rpm 245 kB/s | 33 kB 00:00 2024-09-22T03:06:29.822 INFO:teuthology.orchestra.run.smithi196.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 440 kB/s | 18 kB 00:00 2024-09-22T03:06:29.864 INFO:teuthology.orchestra.run.smithi196.stdout:(5/12): openblas-0.3.26-2.el9.x86_64.rpm 920 kB/s | 38 kB 00:00 2024-09-22T03:06:29.947 INFO:teuthology.orchestra.run.smithi196.stdout:(6/12): ceph-mgr-diskprediction-local-19.1.1-37 16 MB/s | 7.4 MB 00:00 2024-09-22T03:06:30.165 INFO:teuthology.orchestra.run.smithi196.stdout:(7/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 6.3 MB/s | 3.0 MB 00:00 2024-09-22T03:06:30.198 INFO:teuthology.orchestra.run.smithi196.stdout:(8/12): python3-devel-3.9.19-8.el9.x86_64.rpm 977 kB/s | 245 kB 00:00 2024-09-22T03:06:30.282 INFO:teuthology.orchestra.run.smithi196.stdout:(9/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.9 MB/s | 250 kB 00:00 2024-09-22T03:06:30.367 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.1.1-373.g4e45e6b4.el9.noar 11/11 2024-09-22T03:06:30.367 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-grafana-dashboards-2:19.1.1-373.g4e45e6b4.el9 1/11 2024-09-22T03:06:30.367 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-mgr-dashboard-2:19.1.1-373.g4e45e6b4.el9.noar 2/11 2024-09-22T03:06:30.367 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-prometheus-alerts-2:19.1.1-373.g4e45e6b4.el9. 3/11 2024-09-22T03:06:30.367 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 4/11 2024-09-22T03:06:30.367 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 5/11 2024-09-22T03:06:30.368 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 6/11 2024-09-22T03:06:30.368 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 7/11 2024-09-22T03:06:30.368 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 8/11 2024-09-22T03:06:30.368 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/11 2024-09-22T03:06:30.368 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 10/11 2024-09-22T03:06:30.678 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 11/11 2024-09-22T03:06:30.678 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-22T03:06:30.679 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-09-22T03:06:30.679 INFO:teuthology.orchestra.run.smithi153.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-09-22T03:06:30.679 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-grafana-dashboards-2:19.1.1-373.g4e45e6b4.el9.noarch 2024-09-22T03:06:30.679 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-dashboard-2:19.1.1-373.g4e45e6b4.el9.noarch 2024-09-22T03:06:30.679 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-prometheus-alerts-2:19.1.1-373.g4e45e6b4.el9.noarch 2024-09-22T03:06:30.679 INFO:teuthology.orchestra.run.smithi153.stdout: grpc-data-1.46.7-10.el9.noarch 2024-09-22T03:06:30.679 INFO:teuthology.orchestra.run.smithi153.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-09-22T03:06:30.679 INFO:teuthology.orchestra.run.smithi153.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-09-22T03:06:30.679 INFO:teuthology.orchestra.run.smithi153.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-09-22T03:06:30.679 INFO:teuthology.orchestra.run.smithi153.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-09-22T03:06:30.679 INFO:teuthology.orchestra.run.smithi153.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-22T03:06:30.679 INFO:teuthology.orchestra.run.smithi153.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-22T03:06:30.679 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-22T03:06:30.679 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-09-22T03:06:30.912 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-22T03:06:31.493 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:59 ago on Sun 22 Sep 2024 03:04:32 AM UTC. 2024-09-22T03:06:31.523 INFO:teuthology.orchestra.run.smithi196.stdout:(10/12): openblas-openmp-0.3.26-2.el9.x86_64.rp 3.0 MB/s | 4.9 MB 00:01 2024-09-22T03:06:31.627 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-09-22T03:06:31.628 INFO:teuthology.orchestra.run.smithi153.stdout:====================================================================================== 2024-09-22T03:06:31.628 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repository Size 2024-09-22T03:06:31.628 INFO:teuthology.orchestra.run.smithi153.stdout:====================================================================================== 2024-09-22T03:06:31.628 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-09-22T03:06:31.628 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-diskprediction-local noarch 2:19.1.1-373.g4e45e6b4.el9 ceph-noarch 7.4 M 2024-09-22T03:06:31.628 INFO:teuthology.orchestra.run.smithi153.stdout:Installing dependencies: 2024-09-22T03:06:31.629 INFO:teuthology.orchestra.run.smithi153.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-22T03:06:31.629 INFO:teuthology.orchestra.run.smithi153.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-22T03:06:31.629 INFO:teuthology.orchestra.run.smithi153.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-22T03:06:31.629 INFO:teuthology.orchestra.run.smithi153.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-22T03:06:31.629 INFO:teuthology.orchestra.run.smithi153.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-22T03:06:31.629 INFO:teuthology.orchestra.run.smithi153.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-22T03:06:31.629 INFO:teuthology.orchestra.run.smithi153.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-22T03:06:31.629 INFO:teuthology.orchestra.run.smithi153.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-22T03:06:31.629 INFO:teuthology.orchestra.run.smithi153.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-22T03:06:31.629 INFO:teuthology.orchestra.run.smithi153.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-22T03:06:31.629 INFO:teuthology.orchestra.run.smithi153.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-22T03:06:31.629 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-22T03:06:31.629 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-09-22T03:06:31.630 INFO:teuthology.orchestra.run.smithi153.stdout:====================================================================================== 2024-09-22T03:06:31.630 INFO:teuthology.orchestra.run.smithi153.stdout:Install 12 Packages 2024-09-22T03:06:31.630 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-22T03:06:31.630 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 38 M 2024-09-22T03:06:31.630 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 212 M 2024-09-22T03:06:31.630 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-09-22T03:06:31.808 INFO:teuthology.orchestra.run.smithi153.stdout:(1/12): libquadmath-11.5.0-2.el9.x86_64.rpm 1.4 MB/s | 191 kB 00:00 2024-09-22T03:06:31.858 INFO:teuthology.orchestra.run.smithi153.stdout:(2/12): libgfortran-11.5.0-2.el9.x86_64.rpm 4.2 MB/s | 801 kB 00:00 2024-09-22T03:06:31.925 INFO:teuthology.orchestra.run.smithi153.stdout:(3/12): flexiblas-3.0.4-8.el9.x86_64.rpm 279 kB/s | 33 kB 00:00 2024-09-22T03:06:31.967 INFO:teuthology.orchestra.run.smithi153.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 437 kB/s | 18 kB 00:00 2024-09-22T03:06:32.026 INFO:teuthology.orchestra.run.smithi153.stdout:(5/12): openblas-0.3.26-2.el9.x86_64.rpm 658 kB/s | 38 kB 00:00 2024-09-22T03:06:32.084 INFO:teuthology.orchestra.run.smithi153.stdout:(6/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 13 MB/s | 3.0 MB 00:00 2024-09-22T03:06:32.176 INFO:teuthology.orchestra.run.smithi153.stdout:(7/12): ceph-mgr-diskprediction-local-19.1.1-37 15 MB/s | 7.4 MB 00:00 2024-09-22T03:06:32.201 INFO:teuthology.orchestra.run.smithi153.stdout:(8/12): python3-devel-3.9.19-8.el9.x86_64.rpm 2.0 MB/s | 245 kB 00:00 2024-09-22T03:06:32.243 INFO:teuthology.orchestra.run.smithi153.stdout:(9/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 5.9 MB/s | 250 kB 00:00 2024-09-22T03:06:32.307 INFO:teuthology.orchestra.run.smithi196.stdout:(11/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 2.4 MB/s | 5.1 MB 00:02 2024-09-22T03:06:32.335 INFO:teuthology.orchestra.run.smithi153.stdout:(10/12): openblas-openmp-0.3.26-2.el9.x86_64.rp 16 MB/s | 4.9 MB 00:00 2024-09-22T03:06:32.610 INFO:teuthology.orchestra.run.smithi153.stdout:(11/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 44 MB/s | 16 MB 00:00 2024-09-22T03:06:32.694 INFO:teuthology.orchestra.run.smithi153.stdout:(12/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 9.9 MB/s | 5.1 MB 00:00 2024-09-22T03:06:32.695 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-09-22T03:06:32.695 INFO:teuthology.orchestra.run.smithi153.stdout:Total 36 MB/s | 38 MB 00:01 2024-09-22T03:06:33.048 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-09-22T03:06:33.075 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-09-22T03:06:33.075 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-09-22T03:06:33.501 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-09-22T03:06:33.501 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-09-22T03:06:34.257 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-09-22T03:06:34.340 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-22T03:06:34.375 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-22T03:06:34.474 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-22T03:06:34.518 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-22T03:06:34.703 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-22T03:06:34.733 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-22T03:06:34.810 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-22T03:06:35.467 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-22T03:06:35.589 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-22T03:06:36.850 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-22T03:06:37.215 INFO:teuthology.orchestra.run.smithi196.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 2.3 MB/s | 16 MB 00:06 2024-09-22T03:06:37.216 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-09-22T03:06:37.216 INFO:teuthology.orchestra.run.smithi196.stdout:Total 4.9 MB/s | 38 MB 00:07 2024-09-22T03:06:37.578 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-09-22T03:06:37.605 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-09-22T03:06:37.605 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-09-22T03:06:37.874 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-22T03:06:37.899 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-mgr-diskprediction-local-2:19.1.1-373.g4e45e6 12/12 2024-09-22T03:06:38.035 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-09-22T03:06:38.036 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-09-22T03:06:38.789 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-09-22T03:06:38.867 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-22T03:06:38.913 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-22T03:06:39.014 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-22T03:06:39.056 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-22T03:06:39.239 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-22T03:06:39.263 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-22T03:06:39.330 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-22T03:06:39.989 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-22T03:06:40.117 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-22T03:06:40.476 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.1.1-373.g4e45e6 12/12 2024-09-22T03:06:40.477 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.1.1-373.g4e45e6 1/12 2024-09-22T03:06:40.477 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-22T03:06:40.477 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-22T03:06:40.477 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-22T03:06:40.477 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-22T03:06:40.477 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-22T03:06:40.477 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-22T03:06:40.477 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-22T03:06:40.477 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-22T03:06:40.477 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-22T03:06:40.477 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-22T03:06:40.898 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-22T03:06:40.898 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-22T03:06:40.898 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-09-22T03:06:40.898 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-diskprediction-local-2:19.1.1-373.g4e45e6b4.el9.noarch 2024-09-22T03:06:40.898 INFO:teuthology.orchestra.run.smithi153.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-22T03:06:40.898 INFO:teuthology.orchestra.run.smithi153.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-22T03:06:40.898 INFO:teuthology.orchestra.run.smithi153.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-22T03:06:40.898 INFO:teuthology.orchestra.run.smithi153.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-22T03:06:40.898 INFO:teuthology.orchestra.run.smithi153.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-22T03:06:40.898 INFO:teuthology.orchestra.run.smithi153.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-22T03:06:40.898 INFO:teuthology.orchestra.run.smithi153.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-22T03:06:40.899 INFO:teuthology.orchestra.run.smithi153.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-22T03:06:40.899 INFO:teuthology.orchestra.run.smithi153.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-22T03:06:40.899 INFO:teuthology.orchestra.run.smithi153.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-22T03:06:40.899 INFO:teuthology.orchestra.run.smithi153.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-22T03:06:40.899 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-22T03:06:40.899 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-09-22T03:06:41.253 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install ceph-mgr-rook 2024-09-22T03:06:41.376 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-22T03:06:41.839 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:02:09 ago on Sun 22 Sep 2024 03:04:32 AM UTC. 2024-09-22T03:06:41.971 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-09-22T03:06:41.972 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-09-22T03:06:41.972 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repository Size 2024-09-22T03:06:41.972 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-09-22T03:06:41.973 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-09-22T03:06:41.973 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-rook noarch 2:19.1.1-373.g4e45e6b4.el9 ceph-noarch 49 k 2024-09-22T03:06:41.973 INFO:teuthology.orchestra.run.smithi153.stdout:Installing dependencies: 2024-09-22T03:06:41.973 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-22T03:06:41.973 INFO:teuthology.orchestra.run.smithi153.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-22T03:06:41.973 INFO:teuthology.orchestra.run.smithi153.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-22T03:06:41.973 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-22T03:06:41.973 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-22T03:06:41.973 INFO:teuthology.orchestra.run.smithi153.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-22T03:06:41.973 INFO:teuthology.orchestra.run.smithi153.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-22T03:06:41.973 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-22T03:06:41.973 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-22T03:06:41.973 INFO:teuthology.orchestra.run.smithi153.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-22T03:06:41.973 INFO:teuthology.orchestra.run.smithi153.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-22T03:06:41.974 INFO:teuthology.orchestra.run.smithi153.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-22T03:06:41.974 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-22T03:06:41.974 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-09-22T03:06:41.974 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-09-22T03:06:41.974 INFO:teuthology.orchestra.run.smithi153.stdout:Install 13 Packages 2024-09-22T03:06:41.974 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-22T03:06:41.974 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 2.2 M 2024-09-22T03:06:41.974 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 26 M 2024-09-22T03:06:41.975 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-09-22T03:06:42.258 INFO:teuthology.orchestra.run.smithi153.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 208 kB/s | 26 kB 00:00 2024-09-22T03:06:42.283 INFO:teuthology.orchestra.run.smithi153.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 129 kB/s | 19 kB 00:00 2024-09-22T03:06:42.308 INFO:teuthology.orchestra.run.smithi153.stdout:(3/13): ceph-mgr-rook-19.1.1-373.g4e45e6b4.el9. 284 kB/s | 49 kB 00:00 2024-09-22T03:06:42.400 INFO:teuthology.orchestra.run.smithi153.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.3 MB/s | 159 kB 00:00 2024-09-22T03:06:42.413 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-22T03:06:42.426 INFO:teuthology.orchestra.run.smithi153.stdout:(5/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.5 MB/s | 222 kB 00:00 2024-09-22T03:06:42.459 INFO:teuthology.orchestra.run.smithi153.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 917 kB/s | 54 kB 00:00 2024-09-22T03:06:42.493 INFO:teuthology.orchestra.run.smithi153.stdout:(7/13): python3-cachetools-4.2.4-1.el9.noarch.r 481 kB/s | 32 kB 00:00 2024-09-22T03:06:42.518 INFO:teuthology.orchestra.run.smithi153.stdout:(8/13): python3-certifi-2023.05.07-4.el9.noarch 248 kB/s | 14 kB 00:00 2024-09-22T03:06:42.533 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-mgr-diskprediction-local-2:19.1.1-373.g4e45e6 12/12 2024-09-22T03:06:42.543 INFO:teuthology.orchestra.run.smithi153.stdout:(9/13): python3-pyasn1-modules-0.4.8-6.el9.noar 1.2 MB/s | 279 kB 00:00 2024-09-22T03:06:42.568 INFO:teuthology.orchestra.run.smithi153.stdout:(10/13): python3-google-auth-2.34.0-1.el9.noarc 2.9 MB/s | 220 kB 00:00 2024-09-22T03:06:42.602 INFO:teuthology.orchestra.run.smithi153.stdout:(11/13): python3-websocket-client-1.2.3-2.el9.n 3.4 MB/s | 90 kB 00:00 2024-09-22T03:06:42.627 INFO:teuthology.orchestra.run.smithi153.stdout:(12/13): python3-rsa-4.9-2.el9.noarch.rpm 709 kB/s | 59 kB 00:00 2024-09-22T03:06:42.660 INFO:teuthology.orchestra.run.smithi153.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 7.2 MB/s | 1.0 MB 00:00 2024-09-22T03:06:42.663 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-09-22T03:06:42.663 INFO:teuthology.orchestra.run.smithi153.stdout:Total 3.2 MB/s | 2.2 MB 00:00 2024-09-22T03:06:42.725 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-09-22T03:06:42.740 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-09-22T03:06:42.740 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-09-22T03:06:42.829 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-09-22T03:06:42.829 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-09-22T03:06:43.367 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-09-22T03:06:43.524 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-22T03:06:43.587 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-22T03:06:43.642 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-22T03:06:43.679 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-22T03:06:43.721 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-22T03:06:43.823 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-22T03:06:43.924 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-22T03:06:43.981 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-22T03:06:44.695 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-22T03:06:44.752 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-22T03:06:44.786 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-22T03:06:44.834 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-22T03:06:44.856 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-mgr-rook-2:19.1.1-373.g4e45e6b4.el9.noarch 13/13 2024-09-22T03:06:45.002 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.1.1-373.g4e45e6 12/12 2024-09-22T03:06:45.002 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.1.1-373.g4e45e6 1/12 2024-09-22T03:06:45.002 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-22T03:06:45.002 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-22T03:06:45.002 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-22T03:06:45.002 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-22T03:06:45.002 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-22T03:06:45.002 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-22T03:06:45.002 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-22T03:06:45.002 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-22T03:06:45.003 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-22T03:06:45.003 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-22T03:06:45.410 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-22T03:06:45.410 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-22T03:06:45.410 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-09-22T03:06:45.410 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mgr-diskprediction-local-2:19.1.1-373.g4e45e6b4.el9.noarch 2024-09-22T03:06:45.410 INFO:teuthology.orchestra.run.smithi196.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-22T03:06:45.410 INFO:teuthology.orchestra.run.smithi196.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-22T03:06:45.411 INFO:teuthology.orchestra.run.smithi196.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-22T03:06:45.411 INFO:teuthology.orchestra.run.smithi196.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-22T03:06:45.411 INFO:teuthology.orchestra.run.smithi196.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-22T03:06:45.411 INFO:teuthology.orchestra.run.smithi196.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-22T03:06:45.411 INFO:teuthology.orchestra.run.smithi196.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-22T03:06:45.411 INFO:teuthology.orchestra.run.smithi196.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-22T03:06:45.411 INFO:teuthology.orchestra.run.smithi196.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-22T03:06:45.411 INFO:teuthology.orchestra.run.smithi196.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-22T03:06:45.411 INFO:teuthology.orchestra.run.smithi196.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-22T03:06:45.411 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-22T03:06:45.412 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-09-22T03:06:45.705 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-mgr-rook-2:19.1.1-373.g4e45e6b4.el9.noarch 13/13 2024-09-22T03:06:45.705 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-mgr-rook-2:19.1.1-373.g4e45e6b4.el9.noarch 1/13 2024-09-22T03:06:45.705 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-22T03:06:45.705 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-22T03:06:45.705 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-22T03:06:45.705 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-22T03:06:45.705 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-22T03:06:45.706 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-22T03:06:45.706 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-22T03:06:45.706 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-22T03:06:45.706 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-22T03:06:45.706 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-22T03:06:45.706 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-22T03:06:45.779 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install ceph-mgr-rook 2024-09-22T03:06:46.034 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-22T03:06:46.035 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-22T03:06:46.035 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-09-22T03:06:46.035 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-rook-2:19.1.1-373.g4e45e6b4.el9.noarch 2024-09-22T03:06:46.035 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-22T03:06:46.035 INFO:teuthology.orchestra.run.smithi153.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-22T03:06:46.035 INFO:teuthology.orchestra.run.smithi153.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-22T03:06:46.035 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-22T03:06:46.035 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-22T03:06:46.035 INFO:teuthology.orchestra.run.smithi153.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-22T03:06:46.035 INFO:teuthology.orchestra.run.smithi153.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-22T03:06:46.035 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-22T03:06:46.035 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-22T03:06:46.035 INFO:teuthology.orchestra.run.smithi153.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-22T03:06:46.036 INFO:teuthology.orchestra.run.smithi153.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-22T03:06:46.036 INFO:teuthology.orchestra.run.smithi153.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-22T03:06:46.036 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-22T03:06:46.036 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-09-22T03:06:46.277 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install ceph-mgr-cephadm 2024-09-22T03:06:46.374 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:02:15 ago on Sun 22 Sep 2024 03:04:31 AM UTC. 2024-09-22T03:06:46.510 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-09-22T03:06:46.510 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-09-22T03:06:46.510 INFO:teuthology.orchestra.run.smithi196.stdout: Package Arch Version Repository Size 2024-09-22T03:06:46.510 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-09-22T03:06:46.510 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-09-22T03:06:46.510 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mgr-rook noarch 2:19.1.1-373.g4e45e6b4.el9 ceph-noarch 49 k 2024-09-22T03:06:46.510 INFO:teuthology.orchestra.run.smithi196.stdout:Installing dependencies: 2024-09-22T03:06:46.511 INFO:teuthology.orchestra.run.smithi196.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-22T03:06:46.511 INFO:teuthology.orchestra.run.smithi196.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-22T03:06:46.511 INFO:teuthology.orchestra.run.smithi196.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-22T03:06:46.511 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-22T03:06:46.511 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-22T03:06:46.511 INFO:teuthology.orchestra.run.smithi196.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-22T03:06:46.511 INFO:teuthology.orchestra.run.smithi196.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-22T03:06:46.511 INFO:teuthology.orchestra.run.smithi196.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-22T03:06:46.511 INFO:teuthology.orchestra.run.smithi196.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-22T03:06:46.511 INFO:teuthology.orchestra.run.smithi196.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-22T03:06:46.511 INFO:teuthology.orchestra.run.smithi196.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-22T03:06:46.512 INFO:teuthology.orchestra.run.smithi196.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-22T03:06:46.512 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-22T03:06:46.512 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-09-22T03:06:46.512 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-09-22T03:06:46.512 INFO:teuthology.orchestra.run.smithi196.stdout:Install 13 Packages 2024-09-22T03:06:46.512 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-22T03:06:46.512 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 2.2 M 2024-09-22T03:06:46.513 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 26 M 2024-09-22T03:06:46.513 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-09-22T03:06:46.855 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:02:14 ago on Sun 22 Sep 2024 03:04:32 AM UTC. 2024-09-22T03:06:46.987 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-09-22T03:06:46.988 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-09-22T03:06:46.988 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repository Size 2024-09-22T03:06:46.988 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-09-22T03:06:46.988 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-09-22T03:06:46.988 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-cephadm noarch 2:19.1.1-373.g4e45e6b4.el9 ceph-noarch 146 k 2024-09-22T03:06:46.988 INFO:teuthology.orchestra.run.smithi153.stdout:Installing dependencies: 2024-09-22T03:06:46.988 INFO:teuthology.orchestra.run.smithi153.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-22T03:06:46.988 INFO:teuthology.orchestra.run.smithi153.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-22T03:06:46.988 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-22T03:06:46.988 INFO:teuthology.orchestra.run.smithi153.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-22T03:06:46.988 INFO:teuthology.orchestra.run.smithi153.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-22T03:06:46.989 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-22T03:06:46.989 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-09-22T03:06:46.989 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-09-22T03:06:46.989 INFO:teuthology.orchestra.run.smithi153.stdout:Install 6 Packages 2024-09-22T03:06:46.989 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-22T03:06:46.989 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 7.0 M 2024-09-22T03:06:46.989 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 33 M 2024-09-22T03:06:46.989 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-09-22T03:06:47.030 INFO:teuthology.orchestra.run.smithi196.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 330 kB/s | 26 kB 00:00 2024-09-22T03:06:47.056 INFO:teuthology.orchestra.run.smithi196.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 186 kB/s | 19 kB 00:00 2024-09-22T03:06:47.114 INFO:teuthology.orchestra.run.smithi196.stdout:(3/13): ceph-mgr-rook-19.1.1-373.g4e45e6b4.el9. 305 kB/s | 49 kB 00:00 2024-09-22T03:06:47.148 INFO:teuthology.orchestra.run.smithi196.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.9 MB/s | 222 kB 00:00 2024-09-22T03:06:47.190 INFO:teuthology.orchestra.run.smithi196.stdout:(5/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.2 MB/s | 159 kB 00:00 2024-09-22T03:06:47.224 INFO:teuthology.orchestra.run.smithi196.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 712 kB/s | 54 kB 00:00 2024-09-22T03:06:47.257 INFO:teuthology.orchestra.run.smithi196.stdout:(7/13): python3-cachetools-4.2.4-1.el9.noarch.r 480 kB/s | 32 kB 00:00 2024-09-22T03:06:47.282 INFO:teuthology.orchestra.run.smithi196.stdout:(8/13): python3-pyasn1-modules-0.4.8-6.el9.noar 1.6 MB/s | 279 kB 00:00 2024-09-22T03:06:47.299 INFO:teuthology.orchestra.run.smithi196.stdout:(9/13): python3-certifi-2023.05.07-4.el9.noarch 191 kB/s | 14 kB 00:00 2024-09-22T03:06:47.326 INFO:teuthology.orchestra.run.smithi196.stdout:(10/13): python3-google-auth-2.34.0-1.el9.noarc 3.3 MB/s | 220 kB 00:00 2024-09-22T03:06:47.358 INFO:teuthology.orchestra.run.smithi196.stdout:(11/13): python3-websocket-client-1.2.3-2.el9.n 3.4 MB/s | 90 kB 00:00 2024-09-22T03:06:47.400 INFO:teuthology.orchestra.run.smithi196.stdout:(12/13): python3-kubernetes-26.1.0-2.el9.noarch 8.7 MB/s | 1.0 MB 00:00 2024-09-22T03:06:47.417 INFO:teuthology.orchestra.run.smithi196.stdout:(13/13): python3-rsa-4.9-2.el9.noarch.rpm 505 kB/s | 59 kB 00:00 2024-09-22T03:06:47.419 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-09-22T03:06:47.419 INFO:teuthology.orchestra.run.smithi196.stdout:Total 2.4 MB/s | 2.2 MB 00:00 2024-09-22T03:06:47.486 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-09-22T03:06:47.500 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-09-22T03:06:47.501 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-09-22T03:06:47.589 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-09-22T03:06:47.589 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-09-22T03:06:47.654 INFO:teuthology.orchestra.run.smithi153.stdout:(1/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 1.3 MB/s | 248 kB 00:00 2024-09-22T03:06:47.679 INFO:teuthology.orchestra.run.smithi153.stdout:(2/6): ceph-mgr-cephadm-19.1.1-373.g4e45e6b4.el 673 kB/s | 146 kB 00:00 2024-09-22T03:06:47.738 INFO:teuthology.orchestra.run.smithi153.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 990 kB/s | 58 kB 00:00 2024-09-22T03:06:47.771 INFO:teuthology.orchestra.run.smithi153.stdout:(4/6): python3-typing-extensions-4.12.2-2.el9.n 2.2 MB/s | 74 kB 00:00 2024-09-22T03:06:47.796 INFO:teuthology.orchestra.run.smithi153.stdout:(5/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 3.8 MB/s | 548 kB 00:00 2024-09-22T03:06:47.880 INFO:teuthology.orchestra.run.smithi153.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 14 MB/s | 6.0 MB 00:00 2024-09-22T03:06:47.882 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-09-22T03:06:47.883 INFO:teuthology.orchestra.run.smithi153.stdout:Total 7.8 MB/s | 7.0 MB 00:00 2024-09-22T03:06:47.953 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-09-22T03:06:47.964 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-09-22T03:06:47.964 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-09-22T03:06:48.064 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-09-22T03:06:48.066 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-09-22T03:06:48.107 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-09-22T03:06:48.262 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-22T03:06:48.317 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-22T03:06:48.362 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-09-22T03:06:48.373 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-22T03:06:48.419 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-22T03:06:48.460 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-22T03:06:48.528 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-22T03:06:48.562 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-22T03:06:48.581 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-22T03:06:48.664 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-22T03:06:48.720 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-22T03:06:49.106 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-22T03:06:49.193 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-22T03:06:49.256 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-22T03:06:49.286 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-mgr-cephadm-2:19.1.1-373.g4e45e6b4.el9.noarch 6/6 2024-09-22T03:06:49.438 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-22T03:06:49.485 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-22T03:06:49.518 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-22T03:06:49.566 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-22T03:06:49.588 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-mgr-rook-2:19.1.1-373.g4e45e6b4.el9.noarch 13/13 2024-09-22T03:06:50.024 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.1.1-373.g4e45e6b4.el9.noarch 6/6 2024-09-22T03:06:50.024 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-mgr-cephadm-2:19.1.1-373.g4e45e6b4.el9.noarch 1/6 2024-09-22T03:06:50.024 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-22T03:06:50.024 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-22T03:06:50.025 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-22T03:06:50.025 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-22T03:06:50.324 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-22T03:06:50.324 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-22T03:06:50.324 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-09-22T03:06:50.324 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-cephadm-2:19.1.1-373.g4e45e6b4.el9.noarch 2024-09-22T03:06:50.324 INFO:teuthology.orchestra.run.smithi153.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-22T03:06:50.324 INFO:teuthology.orchestra.run.smithi153.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-22T03:06:50.324 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-22T03:06:50.325 INFO:teuthology.orchestra.run.smithi153.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-22T03:06:50.325 INFO:teuthology.orchestra.run.smithi153.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-22T03:06:50.325 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-22T03:06:50.325 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-09-22T03:06:50.352 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-mgr-rook-2:19.1.1-373.g4e45e6b4.el9.noarch 13/13 2024-09-22T03:06:50.352 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-mgr-rook-2:19.1.1-373.g4e45e6b4.el9.noarch 1/13 2024-09-22T03:06:50.352 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-22T03:06:50.353 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-22T03:06:50.353 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-22T03:06:50.353 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-22T03:06:50.353 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-22T03:06:50.353 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-22T03:06:50.353 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-22T03:06:50.353 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-22T03:06:50.353 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-22T03:06:50.353 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-22T03:06:50.353 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-22T03:06:50.561 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install ceph-fuse 2024-09-22T03:06:50.700 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-22T03:06:50.700 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-22T03:06:50.701 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-09-22T03:06:50.701 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mgr-rook-2:19.1.1-373.g4e45e6b4.el9.noarch 2024-09-22T03:06:50.701 INFO:teuthology.orchestra.run.smithi196.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-22T03:06:50.701 INFO:teuthology.orchestra.run.smithi196.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-22T03:06:50.701 INFO:teuthology.orchestra.run.smithi196.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-22T03:06:50.701 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-22T03:06:50.701 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-22T03:06:50.701 INFO:teuthology.orchestra.run.smithi196.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-22T03:06:50.701 INFO:teuthology.orchestra.run.smithi196.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-22T03:06:50.701 INFO:teuthology.orchestra.run.smithi196.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-22T03:06:50.701 INFO:teuthology.orchestra.run.smithi196.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-22T03:06:50.702 INFO:teuthology.orchestra.run.smithi196.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-22T03:06:50.702 INFO:teuthology.orchestra.run.smithi196.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-22T03:06:50.702 INFO:teuthology.orchestra.run.smithi196.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-22T03:06:50.702 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-22T03:06:50.702 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-09-22T03:06:50.990 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install ceph-mgr-cephadm 2024-09-22T03:06:51.141 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:02:19 ago on Sun 22 Sep 2024 03:04:32 AM UTC. 2024-09-22T03:06:51.274 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-09-22T03:06:51.274 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-09-22T03:06:51.275 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repository Size 2024-09-22T03:06:51.275 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-09-22T03:06:51.275 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-09-22T03:06:51.275 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-fuse x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 868 k 2024-09-22T03:06:51.275 INFO:teuthology.orchestra.run.smithi153.stdout:Installing dependencies: 2024-09-22T03:06:51.275 INFO:teuthology.orchestra.run.smithi153.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-22T03:06:51.275 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-22T03:06:51.275 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-09-22T03:06:51.275 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-09-22T03:06:51.275 INFO:teuthology.orchestra.run.smithi153.stdout:Install 2 Packages 2024-09-22T03:06:51.275 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-22T03:06:51.276 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 948 k 2024-09-22T03:06:51.276 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 3.1 M 2024-09-22T03:06:51.276 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-09-22T03:06:51.573 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:02:20 ago on Sun 22 Sep 2024 03:04:31 AM UTC. 2024-09-22T03:06:51.590 INFO:teuthology.orchestra.run.smithi153.stdout:(1/2): ceph-fuse-19.1.1-373.g4e45e6b4.el9.x86_6 3.2 MB/s | 868 kB 00:00 2024-09-22T03:06:51.705 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-09-22T03:06:51.706 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-09-22T03:06:51.706 INFO:teuthology.orchestra.run.smithi196.stdout: Package Arch Version Repository Size 2024-09-22T03:06:51.706 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-09-22T03:06:51.706 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-09-22T03:06:51.706 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mgr-cephadm noarch 2:19.1.1-373.g4e45e6b4.el9 ceph-noarch 146 k 2024-09-22T03:06:51.707 INFO:teuthology.orchestra.run.smithi196.stdout:Installing dependencies: 2024-09-22T03:06:51.707 INFO:teuthology.orchestra.run.smithi196.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-22T03:06:51.707 INFO:teuthology.orchestra.run.smithi196.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-22T03:06:51.707 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-22T03:06:51.707 INFO:teuthology.orchestra.run.smithi196.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-22T03:06:51.707 INFO:teuthology.orchestra.run.smithi196.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-22T03:06:51.707 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-22T03:06:51.707 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-09-22T03:06:51.707 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-09-22T03:06:51.707 INFO:teuthology.orchestra.run.smithi196.stdout:Install 6 Packages 2024-09-22T03:06:51.707 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-22T03:06:51.708 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 7.0 M 2024-09-22T03:06:51.708 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 33 M 2024-09-22T03:06:51.708 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-09-22T03:06:52.056 INFO:teuthology.orchestra.run.smithi153.stdout:(2/2): fuse-2.9.9-16.el9.x86_64.rpm 109 kB/s | 80 kB 00:00 2024-09-22T03:06:52.057 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-09-22T03:06:52.058 INFO:teuthology.orchestra.run.smithi153.stdout:Total 1.2 MB/s | 948 kB 00:00 2024-09-22T03:06:52.071 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-09-22T03:06:52.094 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-09-22T03:06:52.094 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-09-22T03:06:52.102 INFO:teuthology.orchestra.run.smithi196.stdout:(1/6): ceph-mgr-cephadm-19.1.1-373.g4e45e6b4.el 692 kB/s | 146 kB 00:00 2024-09-22T03:06:52.169 INFO:teuthology.orchestra.run.smithi196.stdout:(2/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 8.0 MB/s | 548 kB 00:00 2024-09-22T03:06:52.175 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-09-22T03:06:52.176 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-09-22T03:06:52.194 INFO:teuthology.orchestra.run.smithi196.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 2.3 MB/s | 58 kB 00:00 2024-09-22T03:06:52.220 INFO:teuthology.orchestra.run.smithi196.stdout:(4/6): python3-typing-extensions-4.12.2-2.el9.n 2.9 MB/s | 74 kB 00:00 2024-09-22T03:06:52.287 INFO:teuthology.orchestra.run.smithi196.stdout:(5/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 627 kB/s | 248 kB 00:00 2024-09-22T03:06:52.365 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-09-22T03:06:52.508 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-22T03:06:52.572 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-fuse-2:19.1.1-373.g4e45e6b4.el9.x86_64 2/2 2024-09-22T03:06:53.283 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-fuse-2:19.1.1-373.g4e45e6b4.el9.x86_64 2/2 2024-09-22T03:06:53.283 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-fuse-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/2 2024-09-22T03:06:53.591 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-22T03:06:53.591 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-22T03:06:53.591 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-09-22T03:06:53.591 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-fuse-2:19.1.1-373.g4e45e6b4.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-22T03:06:53.591 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-22T03:06:53.591 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-09-22T03:06:53.884 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install ceph-volume 2024-09-22T03:06:54.482 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:02:22 ago on Sun 22 Sep 2024 03:04:32 AM UTC. 2024-09-22T03:06:54.621 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-09-22T03:06:54.621 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-09-22T03:06:54.621 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repository Size 2024-09-22T03:06:54.621 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-09-22T03:06:54.622 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-09-22T03:06:54.622 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-volume noarch 2:19.1.1-373.g4e45e6b4.el9 ceph-noarch 285 k 2024-09-22T03:06:54.622 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-22T03:06:54.622 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-09-22T03:06:54.622 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-09-22T03:06:54.622 INFO:teuthology.orchestra.run.smithi153.stdout:Install 1 Package 2024-09-22T03:06:54.622 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-22T03:06:54.623 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 285 k 2024-09-22T03:06:54.623 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 1.3 M 2024-09-22T03:06:54.623 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-09-22T03:06:54.852 INFO:teuthology.orchestra.run.smithi153.stdout:ceph-volume-19.1.1-373.g4e45e6b4.el9.noarch.rpm 1.2 MB/s | 285 kB 00:00 2024-09-22T03:06:54.852 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-09-22T03:06:54.853 INFO:teuthology.orchestra.run.smithi153.stdout:Total 1.2 MB/s | 285 kB 00:00 2024-09-22T03:06:54.853 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-09-22T03:06:54.861 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-09-22T03:06:54.861 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-09-22T03:06:54.870 INFO:teuthology.orchestra.run.smithi196.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 2.0 MB/s | 6.0 MB 00:02 2024-09-22T03:06:54.871 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-09-22T03:06:54.872 INFO:teuthology.orchestra.run.smithi196.stdout:Total 2.2 MB/s | 7.0 MB 00:03 2024-09-22T03:06:54.892 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-09-22T03:06:54.892 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-09-22T03:06:54.943 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-09-22T03:06:54.954 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-09-22T03:06:54.954 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-09-22T03:06:55.053 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-09-22T03:06:55.054 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-09-22T03:06:55.115 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-09-22T03:06:55.187 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-volume-2:19.1.1-373.g4e45e6b4.el9.noarch 1/1 2024-09-22T03:06:55.203 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-volume-2:19.1.1-373.g4e45e6b4.el9.noarch 1/1 2024-09-22T03:06:55.203 INFO:teuthology.orchestra.run.smithi153.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-22T03:06:55.203 INFO:teuthology.orchestra.run.smithi153.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-22T03:06:55.203 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-22T03:06:55.360 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-09-22T03:06:55.542 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-22T03:06:55.588 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-22T03:06:56.106 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-22T03:06:56.199 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-22T03:06:56.259 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-volume-2:19.1.1-373.g4e45e6b4.el9.noarch 1/1 2024-09-22T03:06:56.259 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-22T03:06:56.259 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-09-22T03:06:56.259 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-volume-2:19.1.1-373.g4e45e6b4.el9.noarch 2024-09-22T03:06:56.259 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-22T03:06:56.259 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-09-22T03:06:56.262 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-22T03:06:56.285 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-mgr-cephadm-2:19.1.1-373.g4e45e6b4.el9.noarch 6/6 2024-09-22T03:06:56.561 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install librados-devel 2024-09-22T03:06:57.055 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.1.1-373.g4e45e6b4.el9.noarch 6/6 2024-09-22T03:06:57.055 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-mgr-cephadm-2:19.1.1-373.g4e45e6b4.el9.noarch 1/6 2024-09-22T03:06:57.055 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-22T03:06:57.055 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-22T03:06:57.056 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-22T03:06:57.056 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-22T03:06:57.151 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:02:25 ago on Sun 22 Sep 2024 03:04:32 AM UTC. 2024-09-22T03:06:57.284 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-09-22T03:06:57.285 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-09-22T03:06:57.285 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repo Size 2024-09-22T03:06:57.285 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-09-22T03:06:57.285 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-09-22T03:06:57.285 INFO:teuthology.orchestra.run.smithi153.stdout: librados-devel x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 127 k 2024-09-22T03:06:57.286 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-22T03:06:57.286 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-09-22T03:06:57.286 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-09-22T03:06:57.286 INFO:teuthology.orchestra.run.smithi153.stdout:Install 1 Package 2024-09-22T03:06:57.286 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-22T03:06:57.286 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 127 k 2024-09-22T03:06:57.286 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 457 k 2024-09-22T03:06:57.286 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-09-22T03:06:57.340 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-22T03:06:57.340 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-22T03:06:57.340 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-09-22T03:06:57.340 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mgr-cephadm-2:19.1.1-373.g4e45e6b4.el9.noarch 2024-09-22T03:06:57.340 INFO:teuthology.orchestra.run.smithi196.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-22T03:06:57.340 INFO:teuthology.orchestra.run.smithi196.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-22T03:06:57.340 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-22T03:06:57.340 INFO:teuthology.orchestra.run.smithi196.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-22T03:06:57.340 INFO:teuthology.orchestra.run.smithi196.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-22T03:06:57.340 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-22T03:06:57.341 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-09-22T03:06:57.490 INFO:teuthology.orchestra.run.smithi153.stdout:librados-devel-19.1.1-373.g4e45e6b4.el9.x86_64. 623 kB/s | 127 kB 00:00 2024-09-22T03:06:57.490 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-09-22T03:06:57.491 INFO:teuthology.orchestra.run.smithi153.stdout:Total 618 kB/s | 127 kB 00:00 2024-09-22T03:06:57.491 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-09-22T03:06:57.496 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-09-22T03:06:57.496 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-09-22T03:06:57.559 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-09-22T03:06:57.560 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-09-22T03:06:57.617 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install ceph-fuse 2024-09-22T03:06:57.712 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-09-22T03:06:57.820 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : librados-devel-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/1 2024-09-22T03:06:58.196 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:02:27 ago on Sun 22 Sep 2024 03:04:31 AM UTC. 2024-09-22T03:06:58.266 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: librados-devel-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/1 2024-09-22T03:06:58.330 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-09-22T03:06:58.331 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-09-22T03:06:58.331 INFO:teuthology.orchestra.run.smithi196.stdout: Package Arch Version Repository Size 2024-09-22T03:06:58.331 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-09-22T03:06:58.331 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-09-22T03:06:58.331 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-fuse x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 868 k 2024-09-22T03:06:58.331 INFO:teuthology.orchestra.run.smithi196.stdout:Installing dependencies: 2024-09-22T03:06:58.331 INFO:teuthology.orchestra.run.smithi196.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-22T03:06:58.331 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-22T03:06:58.331 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-09-22T03:06:58.332 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-09-22T03:06:58.332 INFO:teuthology.orchestra.run.smithi196.stdout:Install 2 Packages 2024-09-22T03:06:58.332 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-22T03:06:58.332 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 948 k 2024-09-22T03:06:58.332 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 3.1 M 2024-09-22T03:06:58.332 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-09-22T03:06:58.447 INFO:teuthology.orchestra.run.smithi196.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 849 kB/s | 80 kB 00:00 2024-09-22T03:06:58.491 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : librados-devel-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/1 2024-09-22T03:06:58.491 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-22T03:06:58.491 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-09-22T03:06:58.491 INFO:teuthology.orchestra.run.smithi153.stdout: librados-devel-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T03:06:58.491 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-22T03:06:58.491 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-09-22T03:06:58.647 INFO:teuthology.orchestra.run.smithi196.stdout:(2/2): ceph-fuse-19.1.1-373.g4e45e6b4.el9.x86_6 2.9 MB/s | 868 kB 00:00 2024-09-22T03:06:58.648 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-09-22T03:06:58.648 INFO:teuthology.orchestra.run.smithi196.stdout:Total 2.9 MB/s | 948 kB 00:00 2024-09-22T03:06:58.660 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-09-22T03:06:58.682 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-09-22T03:06:58.682 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-09-22T03:06:58.717 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install libcephfs2 2024-09-22T03:06:58.762 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-09-22T03:06:58.762 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-09-22T03:06:58.961 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-09-22T03:06:59.108 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-22T03:06:59.187 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-fuse-2:19.1.1-373.g4e45e6b4.el9.x86_64 2/2 2024-09-22T03:06:59.308 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:02:27 ago on Sun 22 Sep 2024 03:04:32 AM UTC. 2024-09-22T03:06:59.400 INFO:teuthology.orchestra.run.smithi153.stdout:Package libcephfs2-2:19.1.1-373.g4e45e6b4.el9.x86_64 is already installed. 2024-09-22T03:06:59.446 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-09-22T03:06:59.447 INFO:teuthology.orchestra.run.smithi153.stdout:Nothing to do. 2024-09-22T03:06:59.447 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-09-22T03:06:59.522 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install libcephfs-devel 2024-09-22T03:06:59.835 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-fuse-2:19.1.1-373.g4e45e6b4.el9.x86_64 2/2 2024-09-22T03:06:59.835 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-fuse-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/2 2024-09-22T03:07:00.106 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:02:28 ago on Sun 22 Sep 2024 03:04:32 AM UTC. 2024-09-22T03:07:00.150 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-22T03:07:00.150 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-22T03:07:00.150 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-09-22T03:07:00.150 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-fuse-2:19.1.1-373.g4e45e6b4.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-22T03:07:00.150 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-22T03:07:00.150 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-09-22T03:07:00.238 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-09-22T03:07:00.239 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-09-22T03:07:00.240 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repo Size 2024-09-22T03:07:00.240 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-09-22T03:07:00.240 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-09-22T03:07:00.240 INFO:teuthology.orchestra.run.smithi153.stdout: libcephfs-devel x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 32 k 2024-09-22T03:07:00.240 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-22T03:07:00.240 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-09-22T03:07:00.240 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-09-22T03:07:00.240 INFO:teuthology.orchestra.run.smithi153.stdout:Install 1 Package 2024-09-22T03:07:00.240 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-22T03:07:00.240 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 32 k 2024-09-22T03:07:00.240 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 142 k 2024-09-22T03:07:00.240 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-09-22T03:07:00.384 INFO:teuthology.orchestra.run.smithi153.stdout:libcephfs-devel-19.1.1-373.g4e45e6b4.el9.x86_64 221 kB/s | 32 kB 00:00 2024-09-22T03:07:00.385 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-09-22T03:07:00.385 INFO:teuthology.orchestra.run.smithi153.stdout:Total 219 kB/s | 32 kB 00:00 2024-09-22T03:07:00.385 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-09-22T03:07:00.388 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-09-22T03:07:00.388 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-09-22T03:07:00.430 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-09-22T03:07:00.430 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-09-22T03:07:00.467 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install ceph-volume 2024-09-22T03:07:00.520 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-09-22T03:07:00.615 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libcephfs-devel-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/1 2024-09-22T03:07:00.977 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: libcephfs-devel-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/1 2024-09-22T03:07:01.071 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:02:30 ago on Sun 22 Sep 2024 03:04:31 AM UTC. 2024-09-22T03:07:01.206 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-09-22T03:07:01.207 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-09-22T03:07:01.207 INFO:teuthology.orchestra.run.smithi196.stdout: Package Arch Version Repository Size 2024-09-22T03:07:01.207 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-09-22T03:07:01.207 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-09-22T03:07:01.207 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-volume noarch 2:19.1.1-373.g4e45e6b4.el9 ceph-noarch 285 k 2024-09-22T03:07:01.208 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-22T03:07:01.208 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-09-22T03:07:01.208 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-09-22T03:07:01.208 INFO:teuthology.orchestra.run.smithi196.stdout:Install 1 Package 2024-09-22T03:07:01.208 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-22T03:07:01.208 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 285 k 2024-09-22T03:07:01.208 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 1.3 M 2024-09-22T03:07:01.208 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-09-22T03:07:01.221 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libcephfs-devel-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/1 2024-09-22T03:07:01.221 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-22T03:07:01.221 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-09-22T03:07:01.221 INFO:teuthology.orchestra.run.smithi153.stdout: libcephfs-devel-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T03:07:01.221 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-22T03:07:01.222 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-09-22T03:07:01.407 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install librados2 2024-09-22T03:07:01.435 INFO:teuthology.orchestra.run.smithi196.stdout:ceph-volume-19.1.1-373.g4e45e6b4.el9.noarch.rpm 1.2 MB/s | 285 kB 00:00 2024-09-22T03:07:01.436 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-09-22T03:07:01.436 INFO:teuthology.orchestra.run.smithi196.stdout:Total 1.2 MB/s | 285 kB 00:00 2024-09-22T03:07:01.436 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-09-22T03:07:01.445 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-09-22T03:07:01.445 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-09-22T03:07:01.483 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-09-22T03:07:01.483 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-09-22T03:07:01.705 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-09-22T03:07:01.779 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-volume-2:19.1.1-373.g4e45e6b4.el9.noarch 1/1 2024-09-22T03:07:01.795 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-volume-2:19.1.1-373.g4e45e6b4.el9.noarch 1/1 2024-09-22T03:07:01.796 INFO:teuthology.orchestra.run.smithi196.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-22T03:07:01.796 INFO:teuthology.orchestra.run.smithi196.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-22T03:07:01.796 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-22T03:07:01.987 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:02:29 ago on Sun 22 Sep 2024 03:04:32 AM UTC. 2024-09-22T03:07:02.077 INFO:teuthology.orchestra.run.smithi153.stdout:Package librados2-2:19.1.1-373.g4e45e6b4.el9.x86_64 is already installed. 2024-09-22T03:07:02.121 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-09-22T03:07:02.122 INFO:teuthology.orchestra.run.smithi153.stdout:Nothing to do. 2024-09-22T03:07:02.122 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-09-22T03:07:02.193 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install librbd1 2024-09-22T03:07:02.772 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:02:30 ago on Sun 22 Sep 2024 03:04:32 AM UTC. 2024-09-22T03:07:02.804 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-volume-2:19.1.1-373.g4e45e6b4.el9.noarch 1/1 2024-09-22T03:07:02.804 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-22T03:07:02.804 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-09-22T03:07:02.804 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-volume-2:19.1.1-373.g4e45e6b4.el9.noarch 2024-09-22T03:07:02.804 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-22T03:07:02.804 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-09-22T03:07:02.862 INFO:teuthology.orchestra.run.smithi153.stdout:Package librbd1-2:19.1.1-373.g4e45e6b4.el9.x86_64 is already installed. 2024-09-22T03:07:02.906 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-09-22T03:07:02.907 INFO:teuthology.orchestra.run.smithi153.stdout:Nothing to do. 2024-09-22T03:07:02.907 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-09-22T03:07:02.975 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install python3-rados 2024-09-22T03:07:03.153 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install librados-devel 2024-09-22T03:07:03.553 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:02:31 ago on Sun 22 Sep 2024 03:04:32 AM UTC. 2024-09-22T03:07:03.643 INFO:teuthology.orchestra.run.smithi153.stdout:Package python3-rados-2:19.1.1-373.g4e45e6b4.el9.x86_64 is already installed. 2024-09-22T03:07:03.686 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-09-22T03:07:03.687 INFO:teuthology.orchestra.run.smithi153.stdout:Nothing to do. 2024-09-22T03:07:03.687 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-09-22T03:07:03.751 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:02:32 ago on Sun 22 Sep 2024 03:04:31 AM UTC. 2024-09-22T03:07:03.761 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install python3-rgw 2024-09-22T03:07:03.885 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-09-22T03:07:03.885 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-09-22T03:07:03.885 INFO:teuthology.orchestra.run.smithi196.stdout: Package Arch Version Repo Size 2024-09-22T03:07:03.885 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-09-22T03:07:03.885 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-09-22T03:07:03.885 INFO:teuthology.orchestra.run.smithi196.stdout: librados-devel x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 127 k 2024-09-22T03:07:03.885 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-22T03:07:03.885 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-09-22T03:07:03.886 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-09-22T03:07:03.886 INFO:teuthology.orchestra.run.smithi196.stdout:Install 1 Package 2024-09-22T03:07:03.886 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-22T03:07:03.886 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 127 k 2024-09-22T03:07:03.886 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 457 k 2024-09-22T03:07:03.886 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-09-22T03:07:04.107 INFO:teuthology.orchestra.run.smithi196.stdout:librados-devel-19.1.1-373.g4e45e6b4.el9.x86_64. 575 kB/s | 127 kB 00:00 2024-09-22T03:07:04.107 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-09-22T03:07:04.108 INFO:teuthology.orchestra.run.smithi196.stdout:Total 571 kB/s | 127 kB 00:00 2024-09-22T03:07:04.108 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-09-22T03:07:04.113 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-09-22T03:07:04.113 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-09-22T03:07:04.177 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-09-22T03:07:04.177 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-09-22T03:07:04.328 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-09-22T03:07:04.339 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:02:32 ago on Sun 22 Sep 2024 03:04:32 AM UTC. 2024-09-22T03:07:04.430 INFO:teuthology.orchestra.run.smithi153.stdout:Package python3-rgw-2:19.1.1-373.g4e45e6b4.el9.x86_64 is already installed. 2024-09-22T03:07:04.438 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : librados-devel-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/1 2024-09-22T03:07:04.474 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-09-22T03:07:04.475 INFO:teuthology.orchestra.run.smithi153.stdout:Nothing to do. 2024-09-22T03:07:04.475 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-09-22T03:07:04.551 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install python3-cephfs 2024-09-22T03:07:04.886 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: librados-devel-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/1 2024-09-22T03:07:05.122 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:02:33 ago on Sun 22 Sep 2024 03:04:32 AM UTC. 2024-09-22T03:07:05.156 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : librados-devel-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/1 2024-09-22T03:07:05.156 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-22T03:07:05.156 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-09-22T03:07:05.156 INFO:teuthology.orchestra.run.smithi196.stdout: librados-devel-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T03:07:05.156 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-22T03:07:05.156 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-09-22T03:07:05.211 INFO:teuthology.orchestra.run.smithi153.stdout:Package python3-cephfs-2:19.1.1-373.g4e45e6b4.el9.x86_64 is already installed. 2024-09-22T03:07:05.254 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-09-22T03:07:05.255 INFO:teuthology.orchestra.run.smithi153.stdout:Nothing to do. 2024-09-22T03:07:05.255 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-09-22T03:07:05.324 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install python3-rbd 2024-09-22T03:07:05.384 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install libcephfs2 2024-09-22T03:07:05.900 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:02:33 ago on Sun 22 Sep 2024 03:04:32 AM UTC. 2024-09-22T03:07:05.983 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:02:34 ago on Sun 22 Sep 2024 03:04:31 AM UTC. 2024-09-22T03:07:05.989 INFO:teuthology.orchestra.run.smithi153.stdout:Package python3-rbd-2:19.1.1-373.g4e45e6b4.el9.x86_64 is already installed. 2024-09-22T03:07:06.033 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-09-22T03:07:06.034 INFO:teuthology.orchestra.run.smithi153.stdout:Nothing to do. 2024-09-22T03:07:06.034 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-09-22T03:07:06.076 INFO:teuthology.orchestra.run.smithi196.stdout:Package libcephfs2-2:19.1.1-373.g4e45e6b4.el9.x86_64 is already installed. 2024-09-22T03:07:06.100 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install rbd-fuse 2024-09-22T03:07:06.122 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-09-22T03:07:06.123 INFO:teuthology.orchestra.run.smithi196.stdout:Nothing to do. 2024-09-22T03:07:06.123 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-09-22T03:07:06.201 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install libcephfs-devel 2024-09-22T03:07:06.681 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:02:34 ago on Sun 22 Sep 2024 03:04:32 AM UTC. 2024-09-22T03:07:06.782 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:02:35 ago on Sun 22 Sep 2024 03:04:31 AM UTC. 2024-09-22T03:07:06.818 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-09-22T03:07:06.818 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-09-22T03:07:06.818 INFO:teuthology.orchestra.run.smithi153.stdout: Package Architecture Version Repository Size 2024-09-22T03:07:06.818 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-09-22T03:07:06.819 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-09-22T03:07:06.819 INFO:teuthology.orchestra.run.smithi153.stdout: rbd-fuse x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 85 k 2024-09-22T03:07:06.819 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-22T03:07:06.819 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-09-22T03:07:06.819 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-09-22T03:07:06.819 INFO:teuthology.orchestra.run.smithi153.stdout:Install 1 Package 2024-09-22T03:07:06.819 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-22T03:07:06.819 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 85 k 2024-09-22T03:07:06.820 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 256 k 2024-09-22T03:07:06.820 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-09-22T03:07:06.915 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-09-22T03:07:06.916 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-09-22T03:07:06.916 INFO:teuthology.orchestra.run.smithi196.stdout: Package Arch Version Repo Size 2024-09-22T03:07:06.916 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-09-22T03:07:06.916 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-09-22T03:07:06.917 INFO:teuthology.orchestra.run.smithi196.stdout: libcephfs-devel x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 32 k 2024-09-22T03:07:06.917 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-22T03:07:06.917 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-09-22T03:07:06.917 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-09-22T03:07:06.917 INFO:teuthology.orchestra.run.smithi196.stdout:Install 1 Package 2024-09-22T03:07:06.917 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-22T03:07:06.917 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 32 k 2024-09-22T03:07:06.917 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 142 k 2024-09-22T03:07:06.917 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-09-22T03:07:07.012 INFO:teuthology.orchestra.run.smithi153.stdout:rbd-fuse-19.1.1-373.g4e45e6b4.el9.x86_64.rpm 445 kB/s | 85 kB 00:00 2024-09-22T03:07:07.012 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-09-22T03:07:07.012 INFO:teuthology.orchestra.run.smithi153.stdout:Total 442 kB/s | 85 kB 00:00 2024-09-22T03:07:07.013 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-09-22T03:07:07.019 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-09-22T03:07:07.019 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-09-22T03:07:07.061 INFO:teuthology.orchestra.run.smithi196.stdout:libcephfs-devel-19.1.1-373.g4e45e6b4.el9.x86_64 220 kB/s | 32 kB 00:00 2024-09-22T03:07:07.062 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-09-22T03:07:07.062 INFO:teuthology.orchestra.run.smithi196.stdout:Total 218 kB/s | 32 kB 00:00 2024-09-22T03:07:07.062 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-09-22T03:07:07.066 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-09-22T03:07:07.066 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-09-22T03:07:07.083 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-09-22T03:07:07.083 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-09-22T03:07:07.093 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-09-22T03:07:07.094 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-09-22T03:07:07.184 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-09-22T03:07:07.243 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-09-22T03:07:07.299 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : libcephfs-devel-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/1 2024-09-22T03:07:07.367 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : rbd-fuse-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/1 2024-09-22T03:07:07.644 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: libcephfs-devel-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/1 2024-09-22T03:07:07.781 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: rbd-fuse-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/1 2024-09-22T03:07:07.880 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : libcephfs-devel-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/1 2024-09-22T03:07:07.880 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-22T03:07:07.881 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-09-22T03:07:07.881 INFO:teuthology.orchestra.run.smithi196.stdout: libcephfs-devel-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T03:07:07.881 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-22T03:07:07.881 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-09-22T03:07:08.007 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : rbd-fuse-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/1 2024-09-22T03:07:08.007 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-22T03:07:08.008 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-09-22T03:07:08.008 INFO:teuthology.orchestra.run.smithi153.stdout: rbd-fuse-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T03:07:08.008 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-22T03:07:08.008 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-09-22T03:07:08.113 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install librados2 2024-09-22T03:07:08.308 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install rbd-mirror 2024-09-22T03:07:08.695 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:02:37 ago on Sun 22 Sep 2024 03:04:31 AM UTC. 2024-09-22T03:07:08.786 INFO:teuthology.orchestra.run.smithi196.stdout:Package librados2-2:19.1.1-373.g4e45e6b4.el9.x86_64 is already installed. 2024-09-22T03:07:08.830 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-09-22T03:07:08.831 INFO:teuthology.orchestra.run.smithi196.stdout:Nothing to do. 2024-09-22T03:07:08.831 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-09-22T03:07:08.895 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install librbd1 2024-09-22T03:07:08.912 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:02:36 ago on Sun 22 Sep 2024 03:04:32 AM UTC. 2024-09-22T03:07:09.057 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-09-22T03:07:09.057 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-09-22T03:07:09.057 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repo Size 2024-09-22T03:07:09.057 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-09-22T03:07:09.057 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-09-22T03:07:09.057 INFO:teuthology.orchestra.run.smithi153.stdout: rbd-mirror x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 3.1 M 2024-09-22T03:07:09.058 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-22T03:07:09.058 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-09-22T03:07:09.058 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-09-22T03:07:09.058 INFO:teuthology.orchestra.run.smithi153.stdout:Install 1 Package 2024-09-22T03:07:09.058 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-22T03:07:09.058 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 3.1 M 2024-09-22T03:07:09.058 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 14 M 2024-09-22T03:07:09.059 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-09-22T03:07:09.365 INFO:teuthology.orchestra.run.smithi153.stdout:rbd-mirror-19.1.1-373.g4e45e6b4.el9.x86_64.rpm 10 MB/s | 3.1 MB 00:00 2024-09-22T03:07:09.366 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-09-22T03:07:09.366 INFO:teuthology.orchestra.run.smithi153.stdout:Total 9.9 MB/s | 3.1 MB 00:00 2024-09-22T03:07:09.367 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-09-22T03:07:09.376 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-09-22T03:07:09.376 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-09-22T03:07:09.459 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-09-22T03:07:09.459 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-09-22T03:07:09.474 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:02:38 ago on Sun 22 Sep 2024 03:04:31 AM UTC. 2024-09-22T03:07:09.567 INFO:teuthology.orchestra.run.smithi196.stdout:Package librbd1-2:19.1.1-373.g4e45e6b4.el9.x86_64 is already installed. 2024-09-22T03:07:09.611 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-09-22T03:07:09.612 INFO:teuthology.orchestra.run.smithi196.stdout:Nothing to do. 2024-09-22T03:07:09.612 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-09-22T03:07:09.677 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install python3-rados 2024-09-22T03:07:09.911 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-09-22T03:07:09.974 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : rbd-mirror-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/1 2024-09-22T03:07:10.000 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: rbd-mirror-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/1 2024-09-22T03:07:10.001 INFO:teuthology.orchestra.run.smithi153.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-22T03:07:10.001 INFO:teuthology.orchestra.run.smithi153.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-22T03:07:10.001 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-22T03:07:10.001 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-22T03:07:10.001 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-22T03:07:10.254 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:02:39 ago on Sun 22 Sep 2024 03:04:31 AM UTC. 2024-09-22T03:07:10.343 INFO:teuthology.orchestra.run.smithi196.stdout:Package python3-rados-2:19.1.1-373.g4e45e6b4.el9.x86_64 is already installed. 2024-09-22T03:07:10.387 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-09-22T03:07:10.388 INFO:teuthology.orchestra.run.smithi196.stdout:Nothing to do. 2024-09-22T03:07:10.388 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-09-22T03:07:10.454 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install python3-rgw 2024-09-22T03:07:11.034 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:02:40 ago on Sun 22 Sep 2024 03:04:31 AM UTC. 2024-09-22T03:07:11.124 INFO:teuthology.orchestra.run.smithi196.stdout:Package python3-rgw-2:19.1.1-373.g4e45e6b4.el9.x86_64 is already installed. 2024-09-22T03:07:11.131 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : rbd-mirror-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/1 2024-09-22T03:07:11.131 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-22T03:07:11.131 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-09-22T03:07:11.131 INFO:teuthology.orchestra.run.smithi153.stdout: rbd-mirror-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T03:07:11.131 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-22T03:07:11.131 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-09-22T03:07:11.167 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-09-22T03:07:11.169 INFO:teuthology.orchestra.run.smithi196.stdout:Nothing to do. 2024-09-22T03:07:11.169 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-09-22T03:07:11.240 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install python3-cephfs 2024-09-22T03:07:11.359 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install rbd-nbd 2024-09-22T03:07:11.815 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:02:40 ago on Sun 22 Sep 2024 03:04:31 AM UTC. 2024-09-22T03:07:11.905 INFO:teuthology.orchestra.run.smithi196.stdout:Package python3-cephfs-2:19.1.1-373.g4e45e6b4.el9.x86_64 is already installed. 2024-09-22T03:07:11.949 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-09-22T03:07:11.950 INFO:teuthology.orchestra.run.smithi196.stdout:Nothing to do. 2024-09-22T03:07:11.950 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-09-22T03:07:11.962 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:02:39 ago on Sun 22 Sep 2024 03:04:32 AM UTC. 2024-09-22T03:07:12.023 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install python3-rbd 2024-09-22T03:07:12.097 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-09-22T03:07:12.098 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-09-22T03:07:12.098 INFO:teuthology.orchestra.run.smithi153.stdout: Package Architecture Version Repository Size 2024-09-22T03:07:12.098 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-09-22T03:07:12.098 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-09-22T03:07:12.098 INFO:teuthology.orchestra.run.smithi153.stdout: rbd-nbd x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 169 k 2024-09-22T03:07:12.098 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-22T03:07:12.098 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-09-22T03:07:12.098 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-09-22T03:07:12.098 INFO:teuthology.orchestra.run.smithi153.stdout:Install 1 Package 2024-09-22T03:07:12.098 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-22T03:07:12.099 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 169 k 2024-09-22T03:07:12.099 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 540 k 2024-09-22T03:07:12.099 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-09-22T03:07:12.310 INFO:teuthology.orchestra.run.smithi153.stdout:rbd-nbd-19.1.1-373.g4e45e6b4.el9.x86_64.rpm 803 kB/s | 169 kB 00:00 2024-09-22T03:07:12.311 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-09-22T03:07:12.311 INFO:teuthology.orchestra.run.smithi153.stdout:Total 796 kB/s | 169 kB 00:00 2024-09-22T03:07:12.311 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-09-22T03:07:12.319 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-09-22T03:07:12.319 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-09-22T03:07:12.384 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-09-22T03:07:12.384 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-09-22T03:07:12.541 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-09-22T03:07:12.615 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:02:41 ago on Sun 22 Sep 2024 03:04:31 AM UTC. 2024-09-22T03:07:12.642 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : rbd-nbd-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/1 2024-09-22T03:07:12.711 INFO:teuthology.orchestra.run.smithi196.stdout:Package python3-rbd-2:19.1.1-373.g4e45e6b4.el9.x86_64 is already installed. 2024-09-22T03:07:12.757 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-09-22T03:07:12.758 INFO:teuthology.orchestra.run.smithi196.stdout:Nothing to do. 2024-09-22T03:07:12.758 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-09-22T03:07:12.847 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install rbd-fuse 2024-09-22T03:07:13.050 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: rbd-nbd-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/1 2024-09-22T03:07:13.307 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : rbd-nbd-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/1 2024-09-22T03:07:13.307 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-22T03:07:13.307 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-09-22T03:07:13.307 INFO:teuthology.orchestra.run.smithi153.stdout: rbd-nbd-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T03:07:13.307 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-22T03:07:13.307 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-09-22T03:07:13.453 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:02:42 ago on Sun 22 Sep 2024 03:04:31 AM UTC. 2024-09-22T03:07:13.556 DEBUG:teuthology.parallel:result is None 2024-09-22T03:07:13.592 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-09-22T03:07:13.593 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-09-22T03:07:13.593 INFO:teuthology.orchestra.run.smithi196.stdout: Package Architecture Version Repository Size 2024-09-22T03:07:13.593 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-09-22T03:07:13.593 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-09-22T03:07:13.593 INFO:teuthology.orchestra.run.smithi196.stdout: rbd-fuse x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 85 k 2024-09-22T03:07:13.593 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-22T03:07:13.593 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-09-22T03:07:13.593 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-09-22T03:07:13.593 INFO:teuthology.orchestra.run.smithi196.stdout:Install 1 Package 2024-09-22T03:07:13.594 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-22T03:07:13.594 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 85 k 2024-09-22T03:07:13.594 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 256 k 2024-09-22T03:07:13.594 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-09-22T03:07:13.782 INFO:teuthology.orchestra.run.smithi196.stdout:rbd-fuse-19.1.1-373.g4e45e6b4.el9.x86_64.rpm 457 kB/s | 85 kB 00:00 2024-09-22T03:07:13.782 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-09-22T03:07:13.783 INFO:teuthology.orchestra.run.smithi196.stdout:Total 453 kB/s | 85 kB 00:00 2024-09-22T03:07:13.783 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-09-22T03:07:13.790 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-09-22T03:07:13.790 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-09-22T03:07:13.854 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-09-22T03:07:13.854 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-09-22T03:07:14.014 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-09-22T03:07:14.104 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : rbd-fuse-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/1 2024-09-22T03:07:14.491 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: rbd-fuse-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/1 2024-09-22T03:07:14.755 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : rbd-fuse-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/1 2024-09-22T03:07:14.755 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-22T03:07:14.755 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-09-22T03:07:14.755 INFO:teuthology.orchestra.run.smithi196.stdout: rbd-fuse-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T03:07:14.755 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-22T03:07:14.755 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-09-22T03:07:15.043 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install rbd-mirror 2024-09-22T03:07:15.641 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:02:44 ago on Sun 22 Sep 2024 03:04:31 AM UTC. 2024-09-22T03:07:15.780 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-09-22T03:07:15.781 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-09-22T03:07:15.781 INFO:teuthology.orchestra.run.smithi196.stdout: Package Arch Version Repo Size 2024-09-22T03:07:15.781 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-09-22T03:07:15.781 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-09-22T03:07:15.781 INFO:teuthology.orchestra.run.smithi196.stdout: rbd-mirror x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 3.1 M 2024-09-22T03:07:15.781 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-22T03:07:15.781 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-09-22T03:07:15.781 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-09-22T03:07:15.781 INFO:teuthology.orchestra.run.smithi196.stdout:Install 1 Package 2024-09-22T03:07:15.781 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-22T03:07:15.782 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 3.1 M 2024-09-22T03:07:15.782 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 14 M 2024-09-22T03:07:15.782 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-09-22T03:07:16.111 INFO:teuthology.orchestra.run.smithi196.stdout:rbd-mirror-19.1.1-373.g4e45e6b4.el9.x86_64.rpm 9.3 MB/s | 3.1 MB 00:00 2024-09-22T03:07:16.111 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-09-22T03:07:16.112 INFO:teuthology.orchestra.run.smithi196.stdout:Total 9.2 MB/s | 3.1 MB 00:00 2024-09-22T03:07:16.112 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-09-22T03:07:16.121 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-09-22T03:07:16.121 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-09-22T03:07:16.201 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-09-22T03:07:16.201 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-09-22T03:07:16.645 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-09-22T03:07:16.712 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : rbd-mirror-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/1 2024-09-22T03:07:16.737 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: rbd-mirror-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/1 2024-09-22T03:07:16.738 INFO:teuthology.orchestra.run.smithi196.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-22T03:07:16.738 INFO:teuthology.orchestra.run.smithi196.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-22T03:07:16.738 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-22T03:07:16.738 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-22T03:07:16.738 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-22T03:07:17.806 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : rbd-mirror-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/1 2024-09-22T03:07:17.806 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-22T03:07:17.806 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-09-22T03:07:17.807 INFO:teuthology.orchestra.run.smithi196.stdout: rbd-mirror-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T03:07:17.807 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-22T03:07:17.807 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-09-22T03:07:18.073 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install rbd-nbd 2024-09-22T03:07:18.674 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:02:47 ago on Sun 22 Sep 2024 03:04:31 AM UTC. 2024-09-22T03:07:18.808 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-09-22T03:07:18.809 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-09-22T03:07:18.809 INFO:teuthology.orchestra.run.smithi196.stdout: Package Architecture Version Repository Size 2024-09-22T03:07:18.809 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-09-22T03:07:18.809 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-09-22T03:07:18.810 INFO:teuthology.orchestra.run.smithi196.stdout: rbd-nbd x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 169 k 2024-09-22T03:07:18.810 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-22T03:07:18.810 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-09-22T03:07:18.810 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-09-22T03:07:18.810 INFO:teuthology.orchestra.run.smithi196.stdout:Install 1 Package 2024-09-22T03:07:18.810 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-22T03:07:18.810 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 169 k 2024-09-22T03:07:18.810 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 540 k 2024-09-22T03:07:18.810 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-09-22T03:07:19.023 INFO:teuthology.orchestra.run.smithi196.stdout:rbd-nbd-19.1.1-373.g4e45e6b4.el9.x86_64.rpm 797 kB/s | 169 kB 00:00 2024-09-22T03:07:19.024 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-09-22T03:07:19.025 INFO:teuthology.orchestra.run.smithi196.stdout:Total 791 kB/s | 169 kB 00:00 2024-09-22T03:07:19.025 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-09-22T03:07:19.031 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-09-22T03:07:19.031 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-09-22T03:07:19.095 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-09-22T03:07:19.095 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-09-22T03:07:19.246 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-09-22T03:07:19.353 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : rbd-nbd-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/1 2024-09-22T03:07:19.757 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: rbd-nbd-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/1 2024-09-22T03:07:20.021 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : rbd-nbd-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/1 2024-09-22T03:07:20.021 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-22T03:07:20.021 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-09-22T03:07:20.021 INFO:teuthology.orchestra.run.smithi196.stdout: rbd-nbd-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T03:07:20.021 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-22T03:07:20.021 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-09-22T03:07:20.311 DEBUG:teuthology.parallel:result is None 2024-09-22T03:07:20.312 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=4e45e6b490e6cbe082cdf155fea780438ec53fc7 2024-09-22T03:07:20.458 DEBUG:teuthology.orchestra.run.smithi153:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-22T03:07:20.484 INFO:teuthology.orchestra.run.smithi153.stdout:19.1.1-373.g4e45e6b4.el9 2024-09-22T03:07:20.484 INFO:teuthology.packaging:The installed version of ceph is 19.1.1-373.g4e45e6b4.el9 2024-09-22T03:07:20.484 INFO:teuthology.task.install:The correct ceph version 19.1.1-373.g4e45e6b4 is installed. 2024-09-22T03:07:20.487 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=4e45e6b490e6cbe082cdf155fea780438ec53fc7 2024-09-22T03:07:20.628 DEBUG:teuthology.orchestra.run.smithi196:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-22T03:07:20.650 INFO:teuthology.orchestra.run.smithi196.stdout:19.1.1-373.g4e45e6b4.el9 2024-09-22T03:07:20.650 INFO:teuthology.packaging:The installed version of ceph is 19.1.1-373.g4e45e6b4.el9 2024-09-22T03:07:20.650 INFO:teuthology.task.install:The correct ceph version 19.1.1-373.g4e45e6b4 is installed. 2024-09-22T03:07:20.652 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-22T03:07:20.653 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-09-22T03:07:20.653 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-22T03:07:20.683 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-09-22T03:07:20.683 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-22T03:07:20.716 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-22T03:07:20.716 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-09-22T03:07:20.716 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/usr/bin/daemon-helper 2024-09-22T03:07:20.751 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-22T03:07:20.818 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-09-22T03:07:20.818 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd of=/usr/bin/daemon-helper 2024-09-22T03:07:20.843 DEBUG:teuthology.orchestra.run.smithi196:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-22T03:07:20.907 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-22T03:07:20.907 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-09-22T03:07:20.907 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-22T03:07:20.933 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-22T03:07:20.999 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-09-22T03:07:20.999 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-22T03:07:21.024 DEBUG:teuthology.orchestra.run.smithi196:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-22T03:07:21.093 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-22T03:07:21.093 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-09-22T03:07:21.093 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/usr/bin/stdin-killer 2024-09-22T03:07:21.119 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-22T03:07:21.185 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-09-22T03:07:21.185 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd of=/usr/bin/stdin-killer 2024-09-22T03:07:21.210 DEBUG:teuthology.orchestra.run.smithi196:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-22T03:07:21.274 INFO:teuthology.run_tasks:Running task ceph... 2024-09-22T03:07:21.348 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-09-22T03:07:21.349 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod 777 /var/log/ceph 2024-09-22T03:07:21.351 DEBUG:teuthology.orchestra.run.smithi196:> sudo chmod 777 /var/log/ceph 2024-09-22T03:07:21.375 INFO:tasks.ceph:Disabling ceph logrotate... 2024-09-22T03:07:21.375 DEBUG:teuthology.orchestra.run.smithi153:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-22T03:07:21.417 DEBUG:teuthology.orchestra.run.smithi196:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-22T03:07:21.441 INFO:tasks.ceph:Creating extra log directories... 2024-09-22T03:07:21.441 DEBUG:teuthology.orchestra.run.smithi153:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-22T03:07:21.483 DEBUG:teuthology.orchestra.run.smithi196:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-22T03:07:21.521 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-09-22T03:07:21.521 INFO:tasks.ceph:config {'conf': {'osd': {'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd failsafe full ratio': 0.95, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}, 'client.0': {'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc', 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 3}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'osd pool default crimson': True, 'paxos service trim min': 10}, 'osd.0': {'crimson seastar cpu cores': '0-2'}, 'osd.1': {'crimson seastar cpu cores': '3-5'}, 'osd.2': {'crimson seastar cpu cores': '0-2'}, 'osd.3': {'crimson seastar cpu cores': '3-5'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum', 'noscrub', 'nodeep-scrub'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-09-22T03:07:21.521 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/teuthology-2024-09-21_21:56:02-crimson-rados-squid-distro-crimson-smithi/7914725', 'branch': 'squid', 'description': 'crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/radosbench-high-concurrency}', 'email': 'ceph-qa@ceph.com', 'first_in_suite': False, 'job_id': '7914725', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'teuthology-2024-09-21_21:56:02-crimson-rados-squid-distro-crimson-smithi', 'no_nested_subset': False, 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'squid'}, 'ceph': {'conf': {'client.0': {'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc', 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 3}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'osd pool default crimson': True, 'paxos service trim min': 10}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd failsafe full ratio': 0.95, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}, 'osd.0': {'crimson seastar cpu cores': '0-2'}, 'osd.1': {'crimson seastar cpu cores': '3-5'}, 'osd.2': {'crimson seastar cpu cores': '0-2'}, 'osd.3': {'crimson seastar cpu cores': '3-5'}}, 'flavor': 'crimson', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum', 'noscrub', 'nodeep-scrub'], 'sha1': '4e45e6b490e6cbe082cdf155fea780438ec53fc7'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'crimson', 'sha1': '4e45e6b490e6cbe082cdf155fea780438ec53fc7'}}, 'selinux': {'allowlist': ['scontext=system_u:system_r:getty_t:s0']}, 'thrashosds': {'min_in': 4}, 'workunit': {'branch': 'squid', 'env': {'CRIMSON_COMPAT': '1'}, 'sha1': '4e45e6b490e6cbe082cdf155fea780438ec53fc7'}}, 'owner': 'scheduled_teuthology@teuthology', 'priority': 100, 'repo': 'https://git.ceph.com/ceph.git', 'roles': [['mon.a', 'osd.0', 'osd.1', 'client.0', 'node-exporter.a'], ['mgr.x', 'osd.2', 'osd.3', 'client.1', 'prometheus.a', 'node-exporter.b']], 'seed': 1158, 'sha1': '4e45e6b490e6cbe082cdf155fea780438ec53fc7', 'sleep_before_teardown': 0, 'subset': '0/1', 'suite': 'crimson-rados', 'suite_branch': 'squid', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph_4e45e6b490e6cbe082cdf155fea780438ec53fc7/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph.git', 'suite_sha1': '4e45e6b490e6cbe082cdf155fea780438ec53fc7', 'targets': {'smithi153.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFCluRB7uKXFDgmGsvjWZlrq9Js2ZTPKS44fcUbN38nl5K1EMgSiTKj5W7+NM+v5ASunsoio/RzSnHwUHDaBfsk=', 'smithi196.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBErPbLRqq9iv4m0fjlIdE4UAlETUbC2/LEcIBoob52iILGgSQObGaIm5cb+1DtWPSF3r1ufr1Z2to8LCn7vuXgk='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'conf': {'osd': {'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd failsafe full ratio': 0.95, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}, 'client.0': {'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc', 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 3}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'osd pool default crimson': True, 'paxos service trim min': 10}, 'osd.0': {'crimson seastar cpu cores': '0-2'}, 'osd.1': {'crimson seastar cpu cores': '3-5'}, 'osd.2': {'crimson seastar cpu cores': '0-2'}, 'osd.3': {'crimson seastar cpu cores': '3-5'}}, 'flavor': 'crimson', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum', 'noscrub', 'nodeep-scrub'], 'sha1': '4e45e6b490e6cbe082cdf155fea780438ec53fc7', 'cluster': 'ceph'}}, {'thrashosds': {'ceph_objectstore_tool': False, 'chance_inject_pause_short': 0, 'chance_reset_purged_snaps_last': 0, 'chance_thrash_cluster_full': 0, 'chance_thrash_pg_upmap': 0, 'dump_ops_enable': False, 'min_in': 3, 'noscrub_toggle_delay': 0, 'reweight_osd': 0, 'sighup_delay': 0, 'thrash_primary_affinity': False, 'timeout': 2400}}, {'full_sequential': [{'radosbench': {'clients': ['client.0'], 'concurrency': 128, 'size': 8192, 'time': 90}}, {'radosbench': {'clients': ['client.0'], 'concurrency': 128, 'size': 8192, 'time': 90}}, {'radosbench': {'clients': ['client.0'], 'concurrency': 128, 'size': 8192, 'time': 90}}, {'radosbench': {'clients': ['client.0'], 'concurrency': 128, 'size': 8192, 'time': 90}}, {'radosbench': {'clients': ['client.0'], 'concurrency': 128, 'size': 8192, 'time': 90}}, {'radosbench': {'clients': ['client.0'], 'concurrency': 128, 'size': 8192, 'time': 90}}, {'radosbench': {'clients': ['client.0'], 'concurrency': 128, 'size': 8192, 'time': 90}}, {'radosbench': {'clients': ['client.0'], 'concurrency': 128, 'size': 8192, 'time': 90}}]}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a', 'timestamp': '2024-09-21_21:56:02', 'tube': 'smithi', 'user': 'teuthology', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2902'} 2024-09-22T03:07:21.522 DEBUG:teuthology.orchestra.run.smithi153:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-22T03:07:21.548 DEBUG:teuthology.orchestra.run.smithi196:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-22T03:07:21.563 DEBUG:teuthology.orchestra.run.smithi153:> sudo install -d -m0777 -- /var/run/ceph 2024-09-22T03:07:21.604 DEBUG:teuthology.orchestra.run.smithi196:> sudo install -d -m0777 -- /var/run/ceph 2024-09-22T03:07:21.628 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-09-22T03:07:21.628 DEBUG:teuthology.orchestra.run.smithi153:> dd if=/scratch_devs of=/dev/stdout 2024-09-22T03:07:21.682 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-22T03:07:21.682 DEBUG:teuthology.orchestra.run.smithi153:> stat /dev/vg_nvme/lv_1 2024-09-22T03:07:21.737 INFO:teuthology.orchestra.run.smithi153.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-22T03:07:21.738 INFO:teuthology.orchestra.run.smithi153.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-22T03:07:21.738 INFO:teuthology.orchestra.run.smithi153.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-09-22T03:07:21.738 INFO:teuthology.orchestra.run.smithi153.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-22T03:07:21.738 INFO:teuthology.orchestra.run.smithi153.stdout:Context: system_u:object_r:device_t:s0 2024-09-22T03:07:21.738 INFO:teuthology.orchestra.run.smithi153.stdout:Access: 2024-09-22 03:07:10.276447657 +0000 2024-09-22T03:07:21.738 INFO:teuthology.orchestra.run.smithi153.stdout:Modify: 2024-09-22 03:03:51.802382480 +0000 2024-09-22T03:07:21.738 INFO:teuthology.orchestra.run.smithi153.stdout:Change: 2024-09-22 03:03:51.802382480 +0000 2024-09-22T03:07:21.738 INFO:teuthology.orchestra.run.smithi153.stdout: Birth: 2024-09-22 03:03:51.802382480 +0000 2024-09-22T03:07:21.738 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-22T03:07:21.801 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records in 2024-09-22T03:07:21.801 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records out 2024-09-22T03:07:21.801 INFO:teuthology.orchestra.run.smithi153.stderr:512 bytes copied, 0.000221628 s, 2.3 MB/s 2024-09-22T03:07:21.802 DEBUG:teuthology.orchestra.run.smithi153:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-22T03:07:21.858 DEBUG:teuthology.orchestra.run.smithi153:> stat /dev/vg_nvme/lv_2 2024-09-22T03:07:21.914 INFO:teuthology.orchestra.run.smithi153.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-22T03:07:21.914 INFO:teuthology.orchestra.run.smithi153.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-22T03:07:21.914 INFO:teuthology.orchestra.run.smithi153.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-09-22T03:07:21.914 INFO:teuthology.orchestra.run.smithi153.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-22T03:07:21.914 INFO:teuthology.orchestra.run.smithi153.stdout:Context: system_u:object_r:device_t:s0 2024-09-22T03:07:21.915 INFO:teuthology.orchestra.run.smithi153.stdout:Access: 2024-09-22 03:07:10.276447657 +0000 2024-09-22T03:07:21.915 INFO:teuthology.orchestra.run.smithi153.stdout:Modify: 2024-09-22 03:03:52.128377125 +0000 2024-09-22T03:07:21.915 INFO:teuthology.orchestra.run.smithi153.stdout:Change: 2024-09-22 03:03:52.128377125 +0000 2024-09-22T03:07:21.915 INFO:teuthology.orchestra.run.smithi153.stdout: Birth: 2024-09-22 03:03:52.128377125 +0000 2024-09-22T03:07:21.915 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-22T03:07:21.977 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records in 2024-09-22T03:07:21.977 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records out 2024-09-22T03:07:21.978 INFO:teuthology.orchestra.run.smithi153.stderr:512 bytes copied, 0.000195208 s, 2.6 MB/s 2024-09-22T03:07:21.979 DEBUG:teuthology.orchestra.run.smithi153:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-22T03:07:22.035 DEBUG:teuthology.orchestra.run.smithi153:> stat /dev/vg_nvme/lv_3 2024-09-22T03:07:22.089 INFO:teuthology.orchestra.run.smithi153.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-22T03:07:22.089 INFO:teuthology.orchestra.run.smithi153.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-22T03:07:22.089 INFO:teuthology.orchestra.run.smithi153.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-09-22T03:07:22.090 INFO:teuthology.orchestra.run.smithi153.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-22T03:07:22.090 INFO:teuthology.orchestra.run.smithi153.stdout:Context: system_u:object_r:device_t:s0 2024-09-22T03:07:22.090 INFO:teuthology.orchestra.run.smithi153.stdout:Access: 2024-09-22 03:07:10.276447657 +0000 2024-09-22T03:07:22.090 INFO:teuthology.orchestra.run.smithi153.stdout:Modify: 2024-09-22 03:03:52.458371704 +0000 2024-09-22T03:07:22.090 INFO:teuthology.orchestra.run.smithi153.stdout:Change: 2024-09-22 03:03:52.458371704 +0000 2024-09-22T03:07:22.090 INFO:teuthology.orchestra.run.smithi153.stdout: Birth: 2024-09-22 03:03:52.458371704 +0000 2024-09-22T03:07:22.090 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-22T03:07:22.153 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records in 2024-09-22T03:07:22.153 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records out 2024-09-22T03:07:22.153 INFO:teuthology.orchestra.run.smithi153.stderr:512 bytes copied, 0.000207104 s, 2.5 MB/s 2024-09-22T03:07:22.154 DEBUG:teuthology.orchestra.run.smithi153:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-22T03:07:22.210 DEBUG:teuthology.orchestra.run.smithi153:> stat /dev/vg_nvme/lv_4 2024-09-22T03:07:22.266 INFO:teuthology.orchestra.run.smithi153.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-22T03:07:22.266 INFO:teuthology.orchestra.run.smithi153.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-22T03:07:22.266 INFO:teuthology.orchestra.run.smithi153.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-09-22T03:07:22.266 INFO:teuthology.orchestra.run.smithi153.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-22T03:07:22.266 INFO:teuthology.orchestra.run.smithi153.stdout:Context: system_u:object_r:device_t:s0 2024-09-22T03:07:22.266 INFO:teuthology.orchestra.run.smithi153.stdout:Access: 2024-09-22 03:07:10.277447641 +0000 2024-09-22T03:07:22.266 INFO:teuthology.orchestra.run.smithi153.stdout:Modify: 2024-09-22 03:03:52.786366316 +0000 2024-09-22T03:07:22.266 INFO:teuthology.orchestra.run.smithi153.stdout:Change: 2024-09-22 03:03:52.786366316 +0000 2024-09-22T03:07:22.266 INFO:teuthology.orchestra.run.smithi153.stdout: Birth: 2024-09-22 03:03:52.786366316 +0000 2024-09-22T03:07:22.267 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-22T03:07:22.328 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records in 2024-09-22T03:07:22.329 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records out 2024-09-22T03:07:22.329 INFO:teuthology.orchestra.run.smithi153.stderr:512 bytes copied, 0.000188509 s, 2.7 MB/s 2024-09-22T03:07:22.330 DEBUG:teuthology.orchestra.run.smithi153:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-22T03:07:22.387 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2'} 2024-09-22T03:07:22.387 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-09-22T03:07:22.387 DEBUG:teuthology.orchestra.run.smithi196:> dd if=/scratch_devs of=/dev/stdout 2024-09-22T03:07:22.405 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-22T03:07:22.405 DEBUG:teuthology.orchestra.run.smithi196:> stat /dev/vg_nvme/lv_1 2024-09-22T03:07:22.461 INFO:teuthology.orchestra.run.smithi196.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-22T03:07:22.461 INFO:teuthology.orchestra.run.smithi196.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-22T03:07:22.461 INFO:teuthology.orchestra.run.smithi196.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-09-22T03:07:22.461 INFO:teuthology.orchestra.run.smithi196.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-22T03:07:22.461 INFO:teuthology.orchestra.run.smithi196.stdout:Context: system_u:object_r:device_t:s0 2024-09-22T03:07:22.461 INFO:teuthology.orchestra.run.smithi196.stdout:Access: 2024-09-22 03:07:17.001233964 +0000 2024-09-22T03:07:22.462 INFO:teuthology.orchestra.run.smithi196.stdout:Modify: 2024-09-22 03:03:51.801635233 +0000 2024-09-22T03:07:22.462 INFO:teuthology.orchestra.run.smithi196.stdout:Change: 2024-09-22 03:03:51.801635233 +0000 2024-09-22T03:07:22.462 INFO:teuthology.orchestra.run.smithi196.stdout: Birth: 2024-09-22 03:03:51.801635233 +0000 2024-09-22T03:07:22.462 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-22T03:07:22.527 INFO:teuthology.orchestra.run.smithi196.stderr:1+0 records in 2024-09-22T03:07:22.527 INFO:teuthology.orchestra.run.smithi196.stderr:1+0 records out 2024-09-22T03:07:22.527 INFO:teuthology.orchestra.run.smithi196.stderr:512 bytes copied, 0.000232777 s, 2.2 MB/s 2024-09-22T03:07:22.528 DEBUG:teuthology.orchestra.run.smithi196:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-22T03:07:22.586 DEBUG:teuthology.orchestra.run.smithi196:> stat /dev/vg_nvme/lv_2 2024-09-22T03:07:22.642 INFO:teuthology.orchestra.run.smithi196.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-22T03:07:22.642 INFO:teuthology.orchestra.run.smithi196.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-22T03:07:22.642 INFO:teuthology.orchestra.run.smithi196.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-09-22T03:07:22.643 INFO:teuthology.orchestra.run.smithi196.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-22T03:07:22.643 INFO:teuthology.orchestra.run.smithi196.stdout:Context: system_u:object_r:device_t:s0 2024-09-22T03:07:22.643 INFO:teuthology.orchestra.run.smithi196.stdout:Access: 2024-09-22 03:07:17.001233964 +0000 2024-09-22T03:07:22.643 INFO:teuthology.orchestra.run.smithi196.stdout:Modify: 2024-09-22 03:03:52.127628168 +0000 2024-09-22T03:07:22.643 INFO:teuthology.orchestra.run.smithi196.stdout:Change: 2024-09-22 03:03:52.127628168 +0000 2024-09-22T03:07:22.643 INFO:teuthology.orchestra.run.smithi196.stdout: Birth: 2024-09-22 03:03:52.127628168 +0000 2024-09-22T03:07:22.643 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-22T03:07:22.706 INFO:teuthology.orchestra.run.smithi196.stderr:1+0 records in 2024-09-22T03:07:22.706 INFO:teuthology.orchestra.run.smithi196.stderr:1+0 records out 2024-09-22T03:07:22.706 INFO:teuthology.orchestra.run.smithi196.stderr:512 bytes copied, 0.000187975 s, 2.7 MB/s 2024-09-22T03:07:22.707 DEBUG:teuthology.orchestra.run.smithi196:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-22T03:07:22.763 DEBUG:teuthology.orchestra.run.smithi196:> stat /dev/vg_nvme/lv_3 2024-09-22T03:07:22.819 INFO:teuthology.orchestra.run.smithi196.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-22T03:07:22.819 INFO:teuthology.orchestra.run.smithi196.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-22T03:07:22.819 INFO:teuthology.orchestra.run.smithi196.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-09-22T03:07:22.819 INFO:teuthology.orchestra.run.smithi196.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-22T03:07:22.819 INFO:teuthology.orchestra.run.smithi196.stdout:Context: system_u:object_r:device_t:s0 2024-09-22T03:07:22.819 INFO:teuthology.orchestra.run.smithi196.stdout:Access: 2024-09-22 03:07:17.001233964 +0000 2024-09-22T03:07:22.820 INFO:teuthology.orchestra.run.smithi196.stdout:Modify: 2024-09-22 03:03:52.451621148 +0000 2024-09-22T03:07:22.820 INFO:teuthology.orchestra.run.smithi196.stdout:Change: 2024-09-22 03:03:52.451621148 +0000 2024-09-22T03:07:22.820 INFO:teuthology.orchestra.run.smithi196.stdout: Birth: 2024-09-22 03:03:52.451621148 +0000 2024-09-22T03:07:22.820 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-22T03:07:22.883 INFO:teuthology.orchestra.run.smithi196.stderr:1+0 records in 2024-09-22T03:07:22.883 INFO:teuthology.orchestra.run.smithi196.stderr:1+0 records out 2024-09-22T03:07:22.883 INFO:teuthology.orchestra.run.smithi196.stderr:512 bytes copied, 0.000193081 s, 2.7 MB/s 2024-09-22T03:07:22.884 DEBUG:teuthology.orchestra.run.smithi196:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-22T03:07:22.941 DEBUG:teuthology.orchestra.run.smithi196:> stat /dev/vg_nvme/lv_4 2024-09-22T03:07:22.996 INFO:teuthology.orchestra.run.smithi196.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-22T03:07:22.996 INFO:teuthology.orchestra.run.smithi196.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-22T03:07:22.996 INFO:teuthology.orchestra.run.smithi196.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-09-22T03:07:22.996 INFO:teuthology.orchestra.run.smithi196.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-22T03:07:22.996 INFO:teuthology.orchestra.run.smithi196.stdout:Context: system_u:object_r:device_t:s0 2024-09-22T03:07:22.996 INFO:teuthology.orchestra.run.smithi196.stdout:Access: 2024-09-22 03:07:17.002233943 +0000 2024-09-22T03:07:22.996 INFO:teuthology.orchestra.run.smithi196.stdout:Modify: 2024-09-22 03:03:52.778614063 +0000 2024-09-22T03:07:22.997 INFO:teuthology.orchestra.run.smithi196.stdout:Change: 2024-09-22 03:03:52.778614063 +0000 2024-09-22T03:07:22.997 INFO:teuthology.orchestra.run.smithi196.stdout: Birth: 2024-09-22 03:03:52.778614063 +0000 2024-09-22T03:07:22.997 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-22T03:07:23.059 INFO:teuthology.orchestra.run.smithi196.stderr:1+0 records in 2024-09-22T03:07:23.059 INFO:teuthology.orchestra.run.smithi196.stderr:1+0 records out 2024-09-22T03:07:23.059 INFO:teuthology.orchestra.run.smithi196.stderr:512 bytes copied, 0.000288801 s, 1.8 MB/s 2024-09-22T03:07:23.060 DEBUG:teuthology.orchestra.run.smithi196:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-22T03:07:23.117 INFO:tasks.ceph:osd dev map: {'osd.2': '/dev/vg_nvme/lv_1', 'osd.3': '/dev/vg_nvme/lv_2'} 2024-09-22T03:07:23.117 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi153.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2'}, Remote(name='ubuntu@smithi196.front.sepia.ceph.com'): {'osd.2': '/dev/vg_nvme/lv_1', 'osd.3': '/dev/vg_nvme/lv_2'}} 2024-09-22T03:07:23.117 INFO:tasks.ceph:Generating config... 2024-09-22T03:07:23.118 INFO:tasks.ceph:[osd] debug monc = 20 2024-09-22T03:07:23.118 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-09-22T03:07:23.118 INFO:tasks.ceph:[osd] bluestore compression algorithm = snappy 2024-09-22T03:07:23.118 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2024-09-22T03:07:23.118 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-09-22T03:07:23.118 INFO:tasks.ceph:[osd] bluestore rocksdb cf = False 2024-09-22T03:07:23.118 INFO:tasks.ceph:[osd] crimson alien thread cpu cores = 6-7 2024-09-22T03:07:23.118 INFO:tasks.ceph:[osd] crimson osd obc lru size = 10 2024-09-22T03:07:23.119 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-09-22T03:07:23.119 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-09-22T03:07:23.119 INFO:tasks.ceph:[osd] debug ms = 1 2024-09-22T03:07:23.119 INFO:tasks.ceph:[osd] debug osd = 20 2024-09-22T03:07:23.119 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-09-22T03:07:23.119 INFO:tasks.ceph:[osd] err to stderr = True 2024-09-22T03:07:23.119 INFO:tasks.ceph:[osd] log flush on exit = True 2024-09-22T03:07:23.119 INFO:tasks.ceph:[osd] log to file = False 2024-09-22T03:07:23.119 INFO:tasks.ceph:[osd] log to stderr = True 2024-09-22T03:07:23.119 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-09-22T03:07:23.119 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-09-22T03:07:23.119 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-09-22T03:07:23.119 INFO:tasks.ceph:[osd] osd blocked scrub grace period = 3600 2024-09-22T03:07:23.120 INFO:tasks.ceph:[osd] osd debug reject backfill probability = 0.3 2024-09-22T03:07:23.120 INFO:tasks.ceph:[osd] osd delete sleep = 1 2024-09-22T03:07:23.120 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-09-22T03:07:23.120 INFO:tasks.ceph:[osd] osd max backfills = 3 2024-09-22T03:07:23.121 INFO:tasks.ceph:[osd] osd max markdown count = 1000 2024-09-22T03:07:23.121 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-09-22T03:07:23.121 INFO:tasks.ceph:[osd] osd scrub max interval = 120 2024-09-22T03:07:23.121 INFO:tasks.ceph:[osd] osd scrub min interval = 60 2024-09-22T03:07:23.121 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-09-22T03:07:23.122 INFO:tasks.ceph:[osd] osd snap trim sleep = 2 2024-09-22T03:07:23.122 INFO:tasks.ceph:[client.0] debug ms = 1 2024-09-22T03:07:23.122 INFO:tasks.ceph:[client.0] debug objecter = 20 2024-09-22T03:07:23.122 INFO:tasks.ceph:[client.0] debug rados = 20 2024-09-22T03:07:23.122 INFO:tasks.ceph:[global] enable experimental unrecoverable data corrupting features = crimson 2024-09-22T03:07:23.122 INFO:tasks.ceph:[global] ms client mode = crc 2024-09-22T03:07:23.122 INFO:tasks.ceph:[global] ms cluster mode = crc 2024-09-22T03:07:23.122 INFO:tasks.ceph:[global] ms mon client mode = crc 2024-09-22T03:07:23.122 INFO:tasks.ceph:[global] ms mon cluster mode = crc 2024-09-22T03:07:23.122 INFO:tasks.ceph:[global] ms mon service mode = crc 2024-09-22T03:07:23.122 INFO:tasks.ceph:[global] ms service mode = crc 2024-09-22T03:07:23.122 INFO:tasks.ceph:[global] osd_pool_default_min_size = 2 2024-09-22T03:07:23.122 INFO:tasks.ceph:[global] osd_pool_default_size = 3 2024-09-22T03:07:23.123 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-09-22T03:07:23.123 INFO:tasks.ceph:[mgr] debug ms = 1 2024-09-22T03:07:23.123 INFO:tasks.ceph:[mon] debug mon = 20 2024-09-22T03:07:23.123 INFO:tasks.ceph:[mon] debug ms = 1 2024-09-22T03:07:23.123 INFO:tasks.ceph:[mon] debug paxos = 20 2024-09-22T03:07:23.123 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2024-09-22T03:07:23.123 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2024-09-22T03:07:23.123 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2024-09-22T03:07:23.123 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2024-09-22T03:07:23.123 INFO:tasks.ceph:[mon] osd pool default crimson = True 2024-09-22T03:07:23.123 INFO:tasks.ceph:[mon] paxos service trim min = 10 2024-09-22T03:07:23.123 INFO:tasks.ceph:[osd.0] crimson seastar cpu cores = 0-2 2024-09-22T03:07:23.123 INFO:tasks.ceph:[osd.1] crimson seastar cpu cores = 3-5 2024-09-22T03:07:23.123 INFO:tasks.ceph:[osd.2] crimson seastar cpu cores = 0-2 2024-09-22T03:07:23.124 INFO:tasks.ceph:[osd.3] crimson seastar cpu cores = 3-5 2024-09-22T03:07:23.124 INFO:tasks.ceph:Setting up mon.a... 2024-09-22T03:07:23.124 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-09-22T03:07:23.185 INFO:teuthology.orchestra.run.smithi153.stdout:creating /etc/ceph/ceph.keyring 2024-09-22T03:07:23.188 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-09-22T03:07:23.236 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-22T03:07:23.261 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.153')] 2024-09-22T03:07:23.261 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.153', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc', 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 3}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': 120, 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd failsafe full ratio': 0.95, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'osd pool default crimson': True, 'paxos service trim min': 10}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}, 'client.0': {'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'osd.0': {'crimson seastar cpu cores': '0-2'}, 'osd.1': {'crimson seastar cpu cores': '3-5'}, 'osd.2': {'crimson seastar cpu cores': '0-2'}, 'osd.3': {'crimson seastar cpu cores': '3-5'}} 2024-09-22T03:07:23.262 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-09-22T03:07:23.262 DEBUG:teuthology.orchestra.run.smithi153:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-09-22T03:07:23.316 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.153 --print /home/ubuntu/cephtest/ceph.monmap 2024-09-22T03:07:23.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:07:23.400+0000 7f7643f43e80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:07:23.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:07:23.400+0000 7f7643f43e80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:07:23.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:07:23.400+0000 7f7643f43e80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:07:23.403 INFO:teuthology.orchestra.run.smithi153.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-09-22T03:07:23.403 INFO:teuthology.orchestra.run.smithi153.stdout:monmaptool: generated fsid f17abef2-bc08-46d2-abf1-d47a95f0e2ee 2024-09-22T03:07:23.403 INFO:teuthology.orchestra.run.smithi153.stdout:setting min_mon_release = quincy 2024-09-22T03:07:23.403 INFO:teuthology.orchestra.run.smithi153.stdout:epoch 0 2024-09-22T03:07:23.403 INFO:teuthology.orchestra.run.smithi153.stdout:fsid f17abef2-bc08-46d2-abf1-d47a95f0e2ee 2024-09-22T03:07:23.403 INFO:teuthology.orchestra.run.smithi153.stdout:last_changed 2024-09-22T03:07:23.402043+0000 2024-09-22T03:07:23.403 INFO:teuthology.orchestra.run.smithi153.stdout:created 2024-09-22T03:07:23.402043+0000 2024-09-22T03:07:23.403 INFO:teuthology.orchestra.run.smithi153.stdout:min_mon_release 17 (quincy) 2024-09-22T03:07:23.403 INFO:teuthology.orchestra.run.smithi153.stdout:election_strategy: 1 2024-09-22T03:07:23.403 INFO:teuthology.orchestra.run.smithi153.stdout:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-22T03:07:23.404 INFO:teuthology.orchestra.run.smithi153.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (1 monitors) 2024-09-22T03:07:23.405 DEBUG:teuthology.orchestra.run.smithi153:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-09-22T03:07:23.420 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID f17abef2-bc08-46d2-abf1-d47a95f0e2ee... 2024-09-22T03:07:23.420 DEBUG:teuthology.orchestra.run.smithi153:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-22T03:07:23.462 DEBUG:teuthology.orchestra.run.smithi196:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-22T03:07:23.502 INFO:teuthology.orchestra.run.smithi153.stdout:[global] 2024-09-22T03:07:23.502 INFO:teuthology.orchestra.run.smithi153.stdout: chdir = "" 2024-09-22T03:07:23.503 INFO:teuthology.orchestra.run.smithi153.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-22T03:07:23.503 INFO:teuthology.orchestra.run.smithi153.stdout: auth supported = cephx 2024-09-22T03:07:23.503 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-22T03:07:23.503 INFO:teuthology.orchestra.run.smithi153.stdout: filestore xattr use omap = true 2024-09-22T03:07:23.503 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-22T03:07:23.503 INFO:teuthology.orchestra.run.smithi153.stdout: mon clock drift allowed = 1.000 2024-09-22T03:07:23.503 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-22T03:07:23.503 INFO:teuthology.orchestra.run.smithi153.stdout: osd crush chooseleaf type = 0 2024-09-22T03:07:23.503 INFO:teuthology.orchestra.run.smithi153.stdout: auth debug = true 2024-09-22T03:07:23.503 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-22T03:07:23.503 INFO:teuthology.orchestra.run.smithi153.stdout: ms die on old message = true 2024-09-22T03:07:23.503 INFO:teuthology.orchestra.run.smithi153.stdout: ms die on bug = true 2024-09-22T03:07:23.503 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-22T03:07:23.503 INFO:teuthology.orchestra.run.smithi153.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-22T03:07:23.504 INFO:teuthology.orchestra.run.smithi153.stdout: mon pg warn max object skew = 0 2024-09-22T03:07:23.504 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-22T03:07:23.504 INFO:teuthology.orchestra.run.smithi153.stdout: # disable pg_autoscaler by default for new pools 2024-09-22T03:07:23.504 INFO:teuthology.orchestra.run.smithi153.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-22T03:07:23.504 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-22T03:07:23.504 INFO:teuthology.orchestra.run.smithi153.stdout: osd pool default size = 2 2024-09-22T03:07:23.504 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-22T03:07:23.504 INFO:teuthology.orchestra.run.smithi153.stdout: mon osd allow primary affinity = true 2024-09-22T03:07:23.504 INFO:teuthology.orchestra.run.smithi153.stdout: mon osd allow pg remap = true 2024-09-22T03:07:23.504 INFO:teuthology.orchestra.run.smithi153.stdout: mon warn on legacy crush tunables = false 2024-09-22T03:07:23.504 INFO:teuthology.orchestra.run.smithi153.stdout: mon warn on crush straw calc version zero = false 2024-09-22T03:07:23.504 INFO:teuthology.orchestra.run.smithi153.stdout: mon warn on no sortbitwise = false 2024-09-22T03:07:23.504 INFO:teuthology.orchestra.run.smithi153.stdout: mon warn on osd down out interval zero = false 2024-09-22T03:07:23.504 INFO:teuthology.orchestra.run.smithi153.stdout: mon warn on too few osds = false 2024-09-22T03:07:23.505 INFO:teuthology.orchestra.run.smithi153.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-22T03:07:23.505 INFO:teuthology.orchestra.run.smithi153.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-22T03:07:23.505 INFO:teuthology.orchestra.run.smithi153.stdout: mon_allow_pool_size_one = true 2024-09-22T03:07:23.505 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-22T03:07:23.505 INFO:teuthology.orchestra.run.smithi153.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-22T03:07:23.505 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-22T03:07:23.505 INFO:teuthology.orchestra.run.smithi153.stdout: osd default data pool replay window = 5 2024-09-22T03:07:23.505 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-22T03:07:23.505 INFO:teuthology.orchestra.run.smithi153.stdout: mon allow pool delete = true 2024-09-22T03:07:23.505 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-22T03:07:23.505 INFO:teuthology.orchestra.run.smithi153.stdout: mon cluster log file level = debug 2024-09-22T03:07:23.505 INFO:teuthology.orchestra.run.smithi153.stdout: debug asserts on shutdown = true 2024-09-22T03:07:23.505 INFO:teuthology.orchestra.run.smithi153.stdout: mon health detail to clog = false 2024-09-22T03:07:23.505 INFO:teuthology.orchestra.run.smithi153.stdout: mon host = 172.21.15.153 2024-09-22T03:07:23.506 INFO:teuthology.orchestra.run.smithi153.stdout: enable experimental unrecoverable data corrupting features = crimson 2024-09-22T03:07:23.506 INFO:teuthology.orchestra.run.smithi153.stdout: ms client mode = crc 2024-09-22T03:07:23.506 INFO:teuthology.orchestra.run.smithi153.stdout: ms cluster mode = crc 2024-09-22T03:07:23.506 INFO:teuthology.orchestra.run.smithi153.stdout: ms mon client mode = crc 2024-09-22T03:07:23.506 INFO:teuthology.orchestra.run.smithi153.stdout: ms mon cluster mode = crc 2024-09-22T03:07:23.506 INFO:teuthology.orchestra.run.smithi153.stdout: ms mon service mode = crc 2024-09-22T03:07:23.506 INFO:teuthology.orchestra.run.smithi153.stdout: ms service mode = crc 2024-09-22T03:07:23.506 INFO:teuthology.orchestra.run.smithi153.stdout: osd_pool_default_min_size = 2 2024-09-22T03:07:23.506 INFO:teuthology.orchestra.run.smithi153.stdout: osd_pool_default_size = 3 2024-09-22T03:07:23.506 INFO:teuthology.orchestra.run.smithi153.stdout: fsid = f17abef2-bc08-46d2-abf1-d47a95f0e2ee 2024-09-22T03:07:23.506 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-22T03:07:23.506 INFO:teuthology.orchestra.run.smithi153.stdout:[osd] 2024-09-22T03:07:23.506 INFO:teuthology.orchestra.run.smithi153.stdout: osd journal size = 100 2024-09-22T03:07:23.507 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-22T03:07:23.507 INFO:teuthology.orchestra.run.smithi153.stdout: osd scrub load threshold = 5.0 2024-09-22T03:07:23.507 INFO:teuthology.orchestra.run.smithi153.stdout: osd scrub max interval = 120 2024-09-22T03:07:23.507 INFO:teuthology.orchestra.run.smithi153.stdout: osd mclock profile = high_recovery_ops 2024-09-22T03:07:23.507 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-22T03:07:23.507 INFO:teuthology.orchestra.run.smithi153.stdout: osd recover clone overlap = true 2024-09-22T03:07:23.507 INFO:teuthology.orchestra.run.smithi153.stdout: osd recovery max chunk = 1048576 2024-09-22T03:07:23.507 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-22T03:07:23.507 INFO:teuthology.orchestra.run.smithi153.stdout: osd debug shutdown = true 2024-09-22T03:07:23.507 INFO:teuthology.orchestra.run.smithi153.stdout: osd debug op order = true 2024-09-22T03:07:23.507 INFO:teuthology.orchestra.run.smithi153.stdout: osd debug verify stray on activate = true 2024-09-22T03:07:23.507 INFO:teuthology.orchestra.run.smithi153.stdout: osd debug trim objects = true 2024-09-22T03:07:23.507 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-22T03:07:23.507 INFO:teuthology.orchestra.run.smithi153.stdout: osd open classes on start = true 2024-09-22T03:07:23.508 INFO:teuthology.orchestra.run.smithi153.stdout: osd debug pg log writeout = true 2024-09-22T03:07:23.508 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-22T03:07:23.508 INFO:teuthology.orchestra.run.smithi153.stdout: osd deep scrub update digest min age = 30 2024-09-22T03:07:23.508 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-22T03:07:23.508 INFO:teuthology.orchestra.run.smithi153.stdout: osd map max advance = 10 2024-09-22T03:07:23.508 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-22T03:07:23.508 INFO:teuthology.orchestra.run.smithi153.stdout: journal zero on create = true 2024-09-22T03:07:23.508 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-22T03:07:23.508 INFO:teuthology.orchestra.run.smithi153.stdout: filestore ondisk finisher threads = 3 2024-09-22T03:07:23.508 INFO:teuthology.orchestra.run.smithi153.stdout: filestore apply finisher threads = 3 2024-09-22T03:07:23.508 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-22T03:07:23.508 INFO:teuthology.orchestra.run.smithi153.stdout: bdev debug aio = true 2024-09-22T03:07:23.508 INFO:teuthology.orchestra.run.smithi153.stdout: osd debug misdirected ops = true 2024-09-22T03:07:23.508 INFO:teuthology.orchestra.run.smithi153.stdout: debug monc = 20 2024-09-22T03:07:23.509 INFO:teuthology.orchestra.run.smithi153.stdout: bluestore block size = 96636764160 2024-09-22T03:07:23.509 INFO:teuthology.orchestra.run.smithi153.stdout: bluestore compression algorithm = snappy 2024-09-22T03:07:23.509 INFO:teuthology.orchestra.run.smithi153.stdout: bluestore compression mode = aggressive 2024-09-22T03:07:23.509 INFO:teuthology.orchestra.run.smithi153.stdout: bluestore fsck on mount = True 2024-09-22T03:07:23.509 INFO:teuthology.orchestra.run.smithi153.stdout: bluestore rocksdb cf = False 2024-09-22T03:07:23.509 INFO:teuthology.orchestra.run.smithi153.stdout: crimson alien thread cpu cores = 6-7 2024-09-22T03:07:23.509 INFO:teuthology.orchestra.run.smithi153.stdout: crimson osd obc lru size = 10 2024-09-22T03:07:23.509 INFO:teuthology.orchestra.run.smithi153.stdout: debug bluefs = 20 2024-09-22T03:07:23.509 INFO:teuthology.orchestra.run.smithi153.stdout: debug bluestore = 20 2024-09-22T03:07:23.509 INFO:teuthology.orchestra.run.smithi153.stdout: debug ms = 1 2024-09-22T03:07:23.509 INFO:teuthology.orchestra.run.smithi153.stdout: debug osd = 20 2024-09-22T03:07:23.509 INFO:teuthology.orchestra.run.smithi153.stdout: debug rocksdb = 10 2024-09-22T03:07:23.509 INFO:teuthology.orchestra.run.smithi153.stdout: err to stderr = True 2024-09-22T03:07:23.509 INFO:teuthology.orchestra.run.smithi153.stdout: log flush on exit = True 2024-09-22T03:07:23.510 INFO:teuthology.orchestra.run.smithi153.stdout: log to file = False 2024-09-22T03:07:23.510 INFO:teuthology.orchestra.run.smithi153.stdout: log to stderr = True 2024-09-22T03:07:23.510 INFO:teuthology.orchestra.run.smithi153.stdout: mon osd backfillfull_ratio = 0.85 2024-09-22T03:07:23.510 INFO:teuthology.orchestra.run.smithi153.stdout: mon osd full ratio = 0.9 2024-09-22T03:07:23.510 INFO:teuthology.orchestra.run.smithi153.stdout: mon osd nearfull ratio = 0.8 2024-09-22T03:07:23.510 INFO:teuthology.orchestra.run.smithi153.stdout: osd blocked scrub grace period = 3600 2024-09-22T03:07:23.510 INFO:teuthology.orchestra.run.smithi153.stdout: osd debug reject backfill probability = 0.3 2024-09-22T03:07:23.510 INFO:teuthology.orchestra.run.smithi153.stdout: osd delete sleep = 1 2024-09-22T03:07:23.510 INFO:teuthology.orchestra.run.smithi153.stdout: osd failsafe full ratio = 0.95 2024-09-22T03:07:23.510 INFO:teuthology.orchestra.run.smithi153.stdout: osd max backfills = 3 2024-09-22T03:07:23.510 INFO:teuthology.orchestra.run.smithi153.stdout: osd max markdown count = 1000 2024-09-22T03:07:23.510 INFO:teuthology.orchestra.run.smithi153.stdout: osd objectstore = bluestore 2024-09-22T03:07:23.510 INFO:teuthology.orchestra.run.smithi153.stdout: osd scrub min interval = 60 2024-09-22T03:07:23.510 INFO:teuthology.orchestra.run.smithi153.stdout: osd shutdown pgref assert = True 2024-09-22T03:07:23.511 INFO:teuthology.orchestra.run.smithi153.stdout: osd snap trim sleep = 2 2024-09-22T03:07:23.511 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-22T03:07:23.511 INFO:teuthology.orchestra.run.smithi153.stdout:[mgr] 2024-09-22T03:07:23.511 INFO:teuthology.orchestra.run.smithi153.stdout: debug ms = 1 2024-09-22T03:07:23.511 INFO:teuthology.orchestra.run.smithi153.stdout: debug mgr = 20 2024-09-22T03:07:23.511 INFO:teuthology.orchestra.run.smithi153.stdout: debug mon = 20 2024-09-22T03:07:23.511 INFO:teuthology.orchestra.run.smithi153.stdout: debug auth = 20 2024-09-22T03:07:23.511 INFO:teuthology.orchestra.run.smithi153.stdout: mon reweight min pgs per osd = 4 2024-09-22T03:07:23.511 INFO:teuthology.orchestra.run.smithi153.stdout: mon reweight min bytes per osd = 10 2024-09-22T03:07:23.511 INFO:teuthology.orchestra.run.smithi153.stdout: mgr/telemetry/nag = false 2024-09-22T03:07:23.511 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-22T03:07:23.511 INFO:teuthology.orchestra.run.smithi153.stdout:[mon] 2024-09-22T03:07:23.511 INFO:teuthology.orchestra.run.smithi153.stdout: debug ms = 1 2024-09-22T03:07:23.511 INFO:teuthology.orchestra.run.smithi153.stdout: debug mon = 20 2024-09-22T03:07:23.512 INFO:teuthology.orchestra.run.smithi153.stdout: debug paxos = 20 2024-09-22T03:07:23.512 INFO:teuthology.orchestra.run.smithi153.stdout: debug auth = 20 2024-09-22T03:07:23.512 INFO:teuthology.orchestra.run.smithi153.stdout: mon data avail warn = 5 2024-09-22T03:07:23.512 INFO:teuthology.orchestra.run.smithi153.stdout: mon mgr mkfs grace = 240 2024-09-22T03:07:23.512 INFO:teuthology.orchestra.run.smithi153.stdout: mon reweight min pgs per osd = 4 2024-09-22T03:07:23.512 INFO:teuthology.orchestra.run.smithi153.stdout: mon osd reporter subtree level = osd 2024-09-22T03:07:23.512 INFO:teuthology.orchestra.run.smithi153.stdout: mon osd prime pg temp = true 2024-09-22T03:07:23.512 INFO:teuthology.orchestra.run.smithi153.stdout: mon reweight min bytes per osd = 10 2024-09-22T03:07:23.512 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-22T03:07:23.512 INFO:teuthology.orchestra.run.smithi153.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-22T03:07:23.512 INFO:teuthology.orchestra.run.smithi153.stdout: auth mon ticket ttl = 660 # 11m 2024-09-22T03:07:23.512 INFO:teuthology.orchestra.run.smithi153.stdout: auth service ticket ttl = 240 # 4m 2024-09-22T03:07:23.512 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-22T03:07:23.512 INFO:teuthology.orchestra.run.smithi153.stdout: # don't complain about insecure global_id in the test suite 2024-09-22T03:07:23.512 INFO:teuthology.orchestra.run.smithi153.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-22T03:07:23.513 INFO:teuthology.orchestra.run.smithi153.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-22T03:07:23.513 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-22T03:07:23.513 INFO:teuthology.orchestra.run.smithi153.stdout: # 1m isn't quite enough 2024-09-22T03:07:23.513 INFO:teuthology.orchestra.run.smithi153.stdout: mon_down_mkfs_grace = 2m 2024-09-22T03:07:23.513 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-22T03:07:23.513 INFO:teuthology.orchestra.run.smithi153.stdout: mon_warn_on_filestore_osds = false 2024-09-22T03:07:23.513 INFO:teuthology.orchestra.run.smithi153.stdout: mon min osdmap epochs = 50 2024-09-22T03:07:23.513 INFO:teuthology.orchestra.run.smithi153.stdout: mon osdmap full prune interval = 2 2024-09-22T03:07:23.513 INFO:teuthology.orchestra.run.smithi153.stdout: mon osdmap full prune min = 15 2024-09-22T03:07:23.513 INFO:teuthology.orchestra.run.smithi153.stdout: mon osdmap full prune txsize = 2 2024-09-22T03:07:23.513 INFO:teuthology.orchestra.run.smithi153.stdout: osd pool default crimson = True 2024-09-22T03:07:23.513 INFO:teuthology.orchestra.run.smithi153.stdout: paxos service trim min = 10 2024-09-22T03:07:23.513 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-22T03:07:23.513 INFO:teuthology.orchestra.run.smithi153.stdout:[client] 2024-09-22T03:07:23.514 INFO:teuthology.orchestra.run.smithi153.stdout: rgw cache enabled = true 2024-09-22T03:07:23.514 INFO:teuthology.orchestra.run.smithi153.stdout: rgw enable ops log = true 2024-09-22T03:07:23.514 INFO:teuthology.orchestra.run.smithi153.stdout: rgw enable usage log = true 2024-09-22T03:07:23.514 INFO:teuthology.orchestra.run.smithi153.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-22T03:07:23.514 INFO:teuthology.orchestra.run.smithi153.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-22T03:07:23.514 INFO:teuthology.orchestra.run.smithi153.stdout:[mon.a] 2024-09-22T03:07:23.514 INFO:teuthology.orchestra.run.smithi153.stdout:[client.0] 2024-09-22T03:07:23.514 INFO:teuthology.orchestra.run.smithi153.stdout: debug ms = 1 2024-09-22T03:07:23.514 INFO:teuthology.orchestra.run.smithi153.stdout: debug objecter = 20 2024-09-22T03:07:23.514 INFO:teuthology.orchestra.run.smithi153.stdout: debug rados = 20 2024-09-22T03:07:23.514 INFO:teuthology.orchestra.run.smithi153.stdout:[osd.0] 2024-09-22T03:07:23.514 INFO:teuthology.orchestra.run.smithi153.stdout: crimson seastar cpu cores = 0-2 2024-09-22T03:07:23.514 INFO:teuthology.orchestra.run.smithi153.stdout:[osd.1] 2024-09-22T03:07:23.514 INFO:teuthology.orchestra.run.smithi153.stdout: crimson seastar cpu cores = 3-5 2024-09-22T03:07:23.515 INFO:teuthology.orchestra.run.smithi153.stdout:[osd.2] 2024-09-22T03:07:23.515 INFO:teuthology.orchestra.run.smithi153.stdout: crimson seastar cpu cores = 0-2 2024-09-22T03:07:23.515 INFO:teuthology.orchestra.run.smithi153.stdout:[osd.3] 2024-09-22T03:07:23.515 INFO:teuthology.orchestra.run.smithi153.stdout: crimson seastar cpu cores = 3-5 2024-09-22T03:07:23.515 INFO:teuthology.orchestra.run.smithi196.stdout:[global] 2024-09-22T03:07:23.515 INFO:teuthology.orchestra.run.smithi196.stdout: chdir = "" 2024-09-22T03:07:23.515 INFO:teuthology.orchestra.run.smithi196.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-22T03:07:23.515 INFO:teuthology.orchestra.run.smithi196.stdout: auth supported = cephx 2024-09-22T03:07:23.516 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-22T03:07:23.516 INFO:teuthology.orchestra.run.smithi196.stdout: filestore xattr use omap = true 2024-09-22T03:07:23.516 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-22T03:07:23.516 INFO:teuthology.orchestra.run.smithi196.stdout: mon clock drift allowed = 1.000 2024-09-22T03:07:23.516 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-22T03:07:23.516 INFO:teuthology.orchestra.run.smithi196.stdout: osd crush chooseleaf type = 0 2024-09-22T03:07:23.516 INFO:teuthology.orchestra.run.smithi196.stdout: auth debug = true 2024-09-22T03:07:23.516 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-22T03:07:23.516 INFO:teuthology.orchestra.run.smithi196.stdout: ms die on old message = true 2024-09-22T03:07:23.516 INFO:teuthology.orchestra.run.smithi196.stdout: ms die on bug = true 2024-09-22T03:07:23.516 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-22T03:07:23.516 INFO:teuthology.orchestra.run.smithi196.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-22T03:07:23.516 INFO:teuthology.orchestra.run.smithi196.stdout: mon pg warn max object skew = 0 2024-09-22T03:07:23.516 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-22T03:07:23.517 INFO:teuthology.orchestra.run.smithi196.stdout: # disable pg_autoscaler by default for new pools 2024-09-22T03:07:23.517 INFO:teuthology.orchestra.run.smithi196.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-22T03:07:23.517 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-22T03:07:23.517 INFO:teuthology.orchestra.run.smithi196.stdout: osd pool default size = 2 2024-09-22T03:07:23.517 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-22T03:07:23.517 INFO:teuthology.orchestra.run.smithi196.stdout: mon osd allow primary affinity = true 2024-09-22T03:07:23.517 INFO:teuthology.orchestra.run.smithi196.stdout: mon osd allow pg remap = true 2024-09-22T03:07:23.517 INFO:teuthology.orchestra.run.smithi196.stdout: mon warn on legacy crush tunables = false 2024-09-22T03:07:23.517 INFO:teuthology.orchestra.run.smithi196.stdout: mon warn on crush straw calc version zero = false 2024-09-22T03:07:23.517 INFO:teuthology.orchestra.run.smithi196.stdout: mon warn on no sortbitwise = false 2024-09-22T03:07:23.517 INFO:teuthology.orchestra.run.smithi196.stdout: mon warn on osd down out interval zero = false 2024-09-22T03:07:23.517 INFO:teuthology.orchestra.run.smithi196.stdout: mon warn on too few osds = false 2024-09-22T03:07:23.517 INFO:teuthology.orchestra.run.smithi196.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-22T03:07:23.517 INFO:teuthology.orchestra.run.smithi196.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-22T03:07:23.518 INFO:teuthology.orchestra.run.smithi196.stdout: mon_allow_pool_size_one = true 2024-09-22T03:07:23.518 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-22T03:07:23.518 INFO:teuthology.orchestra.run.smithi196.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-22T03:07:23.518 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-22T03:07:23.518 INFO:teuthology.orchestra.run.smithi196.stdout: osd default data pool replay window = 5 2024-09-22T03:07:23.518 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-22T03:07:23.518 INFO:teuthology.orchestra.run.smithi196.stdout: mon allow pool delete = true 2024-09-22T03:07:23.518 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-22T03:07:23.518 INFO:teuthology.orchestra.run.smithi196.stdout: mon cluster log file level = debug 2024-09-22T03:07:23.518 INFO:teuthology.orchestra.run.smithi196.stdout: debug asserts on shutdown = true 2024-09-22T03:07:23.518 INFO:teuthology.orchestra.run.smithi196.stdout: mon health detail to clog = false 2024-09-22T03:07:23.518 INFO:teuthology.orchestra.run.smithi196.stdout: mon host = 172.21.15.153 2024-09-22T03:07:23.518 INFO:teuthology.orchestra.run.smithi196.stdout: enable experimental unrecoverable data corrupting features = crimson 2024-09-22T03:07:23.518 INFO:teuthology.orchestra.run.smithi196.stdout: ms client mode = crc 2024-09-22T03:07:23.519 INFO:teuthology.orchestra.run.smithi196.stdout: ms cluster mode = crc 2024-09-22T03:07:23.519 INFO:teuthology.orchestra.run.smithi196.stdout: ms mon client mode = crc 2024-09-22T03:07:23.519 INFO:teuthology.orchestra.run.smithi196.stdout: ms mon cluster mode = crc 2024-09-22T03:07:23.519 INFO:teuthology.orchestra.run.smithi196.stdout: ms mon service mode = crc 2024-09-22T03:07:23.519 INFO:teuthology.orchestra.run.smithi196.stdout: ms service mode = crc 2024-09-22T03:07:23.519 INFO:teuthology.orchestra.run.smithi196.stdout: osd_pool_default_min_size = 2 2024-09-22T03:07:23.519 INFO:teuthology.orchestra.run.smithi196.stdout: osd_pool_default_size = 3 2024-09-22T03:07:23.519 INFO:teuthology.orchestra.run.smithi196.stdout: fsid = f17abef2-bc08-46d2-abf1-d47a95f0e2ee 2024-09-22T03:07:23.519 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-22T03:07:23.519 INFO:teuthology.orchestra.run.smithi196.stdout:[osd] 2024-09-22T03:07:23.519 INFO:teuthology.orchestra.run.smithi196.stdout: osd journal size = 100 2024-09-22T03:07:23.519 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-22T03:07:23.519 INFO:teuthology.orchestra.run.smithi196.stdout: osd scrub load threshold = 5.0 2024-09-22T03:07:23.519 INFO:teuthology.orchestra.run.smithi196.stdout: osd scrub max interval = 120 2024-09-22T03:07:23.520 INFO:teuthology.orchestra.run.smithi196.stdout: osd mclock profile = high_recovery_ops 2024-09-22T03:07:23.520 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-22T03:07:23.520 INFO:teuthology.orchestra.run.smithi196.stdout: osd recover clone overlap = true 2024-09-22T03:07:23.520 INFO:teuthology.orchestra.run.smithi196.stdout: osd recovery max chunk = 1048576 2024-09-22T03:07:23.520 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-22T03:07:23.520 INFO:teuthology.orchestra.run.smithi196.stdout: osd debug shutdown = true 2024-09-22T03:07:23.520 INFO:teuthology.orchestra.run.smithi196.stdout: osd debug op order = true 2024-09-22T03:07:23.520 INFO:teuthology.orchestra.run.smithi196.stdout: osd debug verify stray on activate = true 2024-09-22T03:07:23.520 INFO:teuthology.orchestra.run.smithi196.stdout: osd debug trim objects = true 2024-09-22T03:07:23.520 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-22T03:07:23.520 INFO:teuthology.orchestra.run.smithi196.stdout: osd open classes on start = true 2024-09-22T03:07:23.520 INFO:teuthology.orchestra.run.smithi196.stdout: osd debug pg log writeout = true 2024-09-22T03:07:23.520 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-22T03:07:23.520 INFO:teuthology.orchestra.run.smithi196.stdout: osd deep scrub update digest min age = 30 2024-09-22T03:07:23.521 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-22T03:07:23.521 INFO:teuthology.orchestra.run.smithi196.stdout: osd map max advance = 10 2024-09-22T03:07:23.521 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-22T03:07:23.521 INFO:teuthology.orchestra.run.smithi196.stdout: journal zero on create = true 2024-09-22T03:07:23.521 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-22T03:07:23.521 INFO:teuthology.orchestra.run.smithi196.stdout: filestore ondisk finisher threads = 3 2024-09-22T03:07:23.521 INFO:teuthology.orchestra.run.smithi196.stdout: filestore apply finisher threads = 3 2024-09-22T03:07:23.521 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-22T03:07:23.521 INFO:teuthology.orchestra.run.smithi196.stdout: bdev debug aio = true 2024-09-22T03:07:23.521 INFO:teuthology.orchestra.run.smithi196.stdout: osd debug misdirected ops = true 2024-09-22T03:07:23.521 INFO:teuthology.orchestra.run.smithi196.stdout: debug monc = 20 2024-09-22T03:07:23.521 INFO:teuthology.orchestra.run.smithi196.stdout: bluestore block size = 96636764160 2024-09-22T03:07:23.521 INFO:teuthology.orchestra.run.smithi196.stdout: bluestore compression algorithm = snappy 2024-09-22T03:07:23.521 INFO:teuthology.orchestra.run.smithi196.stdout: bluestore compression mode = aggressive 2024-09-22T03:07:23.522 INFO:teuthology.orchestra.run.smithi196.stdout: bluestore fsck on mount = True 2024-09-22T03:07:23.522 INFO:teuthology.orchestra.run.smithi196.stdout: bluestore rocksdb cf = False 2024-09-22T03:07:23.522 INFO:teuthology.orchestra.run.smithi196.stdout: crimson alien thread cpu cores = 6-7 2024-09-22T03:07:23.522 INFO:teuthology.orchestra.run.smithi196.stdout: crimson osd obc lru size = 10 2024-09-22T03:07:23.522 INFO:teuthology.orchestra.run.smithi196.stdout: debug bluefs = 20 2024-09-22T03:07:23.522 INFO:teuthology.orchestra.run.smithi196.stdout: debug bluestore = 20 2024-09-22T03:07:23.522 INFO:teuthology.orchestra.run.smithi196.stdout: debug ms = 1 2024-09-22T03:07:23.522 INFO:teuthology.orchestra.run.smithi196.stdout: debug osd = 20 2024-09-22T03:07:23.522 INFO:teuthology.orchestra.run.smithi196.stdout: debug rocksdb = 10 2024-09-22T03:07:23.522 INFO:teuthology.orchestra.run.smithi196.stdout: err to stderr = True 2024-09-22T03:07:23.522 INFO:teuthology.orchestra.run.smithi196.stdout: log flush on exit = True 2024-09-22T03:07:23.522 INFO:teuthology.orchestra.run.smithi196.stdout: log to file = False 2024-09-22T03:07:23.522 INFO:teuthology.orchestra.run.smithi196.stdout: log to stderr = True 2024-09-22T03:07:23.522 INFO:teuthology.orchestra.run.smithi196.stdout: mon osd backfillfull_ratio = 0.85 2024-09-22T03:07:23.522 INFO:teuthology.orchestra.run.smithi196.stdout: mon osd full ratio = 0.9 2024-09-22T03:07:23.523 INFO:teuthology.orchestra.run.smithi196.stdout: mon osd nearfull ratio = 0.8 2024-09-22T03:07:23.523 INFO:teuthology.orchestra.run.smithi196.stdout: osd blocked scrub grace period = 3600 2024-09-22T03:07:23.523 INFO:teuthology.orchestra.run.smithi196.stdout: osd debug reject backfill probability = 0.3 2024-09-22T03:07:23.523 INFO:teuthology.orchestra.run.smithi196.stdout: osd delete sleep = 1 2024-09-22T03:07:23.523 INFO:teuthology.orchestra.run.smithi196.stdout: osd failsafe full ratio = 0.95 2024-09-22T03:07:23.523 INFO:teuthology.orchestra.run.smithi196.stdout: osd max backfills = 3 2024-09-22T03:07:23.523 INFO:teuthology.orchestra.run.smithi196.stdout: osd max markdown count = 1000 2024-09-22T03:07:23.523 INFO:teuthology.orchestra.run.smithi196.stdout: osd objectstore = bluestore 2024-09-22T03:07:23.523 INFO:teuthology.orchestra.run.smithi196.stdout: osd scrub min interval = 60 2024-09-22T03:07:23.523 INFO:teuthology.orchestra.run.smithi196.stdout: osd shutdown pgref assert = True 2024-09-22T03:07:23.523 INFO:teuthology.orchestra.run.smithi196.stdout: osd snap trim sleep = 2 2024-09-22T03:07:23.523 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-22T03:07:23.523 INFO:teuthology.orchestra.run.smithi196.stdout:[mgr] 2024-09-22T03:07:23.523 INFO:teuthology.orchestra.run.smithi196.stdout: debug ms = 1 2024-09-22T03:07:23.524 INFO:teuthology.orchestra.run.smithi196.stdout: debug mgr = 20 2024-09-22T03:07:23.524 INFO:teuthology.orchestra.run.smithi196.stdout: debug mon = 20 2024-09-22T03:07:23.524 INFO:teuthology.orchestra.run.smithi196.stdout: debug auth = 20 2024-09-22T03:07:23.524 INFO:teuthology.orchestra.run.smithi196.stdout: mon reweight min pgs per osd = 4 2024-09-22T03:07:23.524 INFO:teuthology.orchestra.run.smithi196.stdout: mon reweight min bytes per osd = 10 2024-09-22T03:07:23.524 INFO:teuthology.orchestra.run.smithi196.stdout: mgr/telemetry/nag = false 2024-09-22T03:07:23.524 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-22T03:07:23.524 INFO:teuthology.orchestra.run.smithi196.stdout:[mon] 2024-09-22T03:07:23.524 INFO:teuthology.orchestra.run.smithi196.stdout: debug ms = 1 2024-09-22T03:07:23.524 INFO:teuthology.orchestra.run.smithi196.stdout: debug mon = 20 2024-09-22T03:07:23.524 INFO:teuthology.orchestra.run.smithi196.stdout: debug paxos = 20 2024-09-22T03:07:23.524 INFO:teuthology.orchestra.run.smithi196.stdout: debug auth = 20 2024-09-22T03:07:23.524 INFO:teuthology.orchestra.run.smithi196.stdout: mon data avail warn = 5 2024-09-22T03:07:23.524 INFO:teuthology.orchestra.run.smithi196.stdout: mon mgr mkfs grace = 240 2024-09-22T03:07:23.525 INFO:teuthology.orchestra.run.smithi196.stdout: mon reweight min pgs per osd = 4 2024-09-22T03:07:23.525 INFO:teuthology.orchestra.run.smithi196.stdout: mon osd reporter subtree level = osd 2024-09-22T03:07:23.525 INFO:teuthology.orchestra.run.smithi196.stdout: mon osd prime pg temp = true 2024-09-22T03:07:23.525 INFO:teuthology.orchestra.run.smithi196.stdout: mon reweight min bytes per osd = 10 2024-09-22T03:07:23.525 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-22T03:07:23.525 INFO:teuthology.orchestra.run.smithi196.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-22T03:07:23.525 INFO:teuthology.orchestra.run.smithi196.stdout: auth mon ticket ttl = 660 # 11m 2024-09-22T03:07:23.525 INFO:teuthology.orchestra.run.smithi196.stdout: auth service ticket ttl = 240 # 4m 2024-09-22T03:07:23.525 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-22T03:07:23.525 INFO:teuthology.orchestra.run.smithi196.stdout: # don't complain about insecure global_id in the test suite 2024-09-22T03:07:23.525 INFO:teuthology.orchestra.run.smithi196.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-22T03:07:23.525 INFO:teuthology.orchestra.run.smithi196.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-22T03:07:23.525 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-22T03:07:23.525 INFO:teuthology.orchestra.run.smithi196.stdout: # 1m isn't quite enough 2024-09-22T03:07:23.525 INFO:teuthology.orchestra.run.smithi196.stdout: mon_down_mkfs_grace = 2m 2024-09-22T03:07:23.526 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-22T03:07:23.526 INFO:teuthology.orchestra.run.smithi196.stdout: mon_warn_on_filestore_osds = false 2024-09-22T03:07:23.526 INFO:teuthology.orchestra.run.smithi196.stdout: mon min osdmap epochs = 50 2024-09-22T03:07:23.526 INFO:teuthology.orchestra.run.smithi196.stdout: mon osdmap full prune interval = 2 2024-09-22T03:07:23.526 INFO:teuthology.orchestra.run.smithi196.stdout: mon osdmap full prune min = 15 2024-09-22T03:07:23.526 INFO:teuthology.orchestra.run.smithi196.stdout: mon osdmap full prune txsize = 2 2024-09-22T03:07:23.526 INFO:teuthology.orchestra.run.smithi196.stdout: osd pool default crimson = True 2024-09-22T03:07:23.526 INFO:teuthology.orchestra.run.smithi196.stdout: paxos service trim min = 10 2024-09-22T03:07:23.526 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-09-22T03:07:23.526 INFO:teuthology.orchestra.run.smithi196.stdout:[client] 2024-09-22T03:07:23.526 INFO:teuthology.orchestra.run.smithi196.stdout: rgw cache enabled = true 2024-09-22T03:07:23.526 INFO:teuthology.orchestra.run.smithi196.stdout: rgw enable ops log = true 2024-09-22T03:07:23.526 INFO:teuthology.orchestra.run.smithi196.stdout: rgw enable usage log = true 2024-09-22T03:07:23.526 INFO:teuthology.orchestra.run.smithi196.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-22T03:07:23.527 INFO:teuthology.orchestra.run.smithi196.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-22T03:07:23.527 INFO:teuthology.orchestra.run.smithi196.stdout:[mon.a] 2024-09-22T03:07:23.527 INFO:teuthology.orchestra.run.smithi196.stdout:[client.0] 2024-09-22T03:07:23.527 INFO:teuthology.orchestra.run.smithi196.stdout: debug ms = 1 2024-09-22T03:07:23.527 INFO:teuthology.orchestra.run.smithi196.stdout: debug objecter = 20 2024-09-22T03:07:23.527 INFO:teuthology.orchestra.run.smithi196.stdout: debug rados = 20 2024-09-22T03:07:23.527 INFO:teuthology.orchestra.run.smithi196.stdout:[osd.0] 2024-09-22T03:07:23.527 INFO:teuthology.orchestra.run.smithi196.stdout: crimson seastar cpu cores = 0-2 2024-09-22T03:07:23.527 INFO:teuthology.orchestra.run.smithi196.stdout:[osd.1] 2024-09-22T03:07:23.527 INFO:teuthology.orchestra.run.smithi196.stdout: crimson seastar cpu cores = 3-5 2024-09-22T03:07:23.527 INFO:teuthology.orchestra.run.smithi196.stdout:[osd.2] 2024-09-22T03:07:23.527 INFO:teuthology.orchestra.run.smithi196.stdout: crimson seastar cpu cores = 0-2 2024-09-22T03:07:23.527 INFO:teuthology.orchestra.run.smithi196.stdout:[osd.3] 2024-09-22T03:07:23.527 INFO:teuthology.orchestra.run.smithi196.stdout: crimson seastar cpu cores = 3-5 2024-09-22T03:07:23.530 INFO:tasks.ceph:Creating admin key on mon.a... 2024-09-22T03:07:23.531 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-09-22T03:07:23.631 INFO:tasks.ceph:Copying monmap to all nodes... 2024-09-22T03:07:23.631 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-09-22T03:07:23.631 DEBUG:teuthology.orchestra.run.smithi153:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-09-22T03:07:23.646 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-09-22T03:07:23.647 DEBUG:teuthology.orchestra.run.smithi153:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-09-22T03:07:23.701 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi153.front.sepia.ceph.com 2024-09-22T03:07:23.701 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-09-22T03:07:23.701 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-22T03:07:23.701 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-22T03:07:23.776 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-09-22T03:07:23.776 DEBUG:teuthology.orchestra.run.smithi153:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-22T03:07:23.832 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi196.front.sepia.ceph.com 2024-09-22T03:07:23.832 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-09-22T03:07:23.832 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-22T03:07:23.832 DEBUG:teuthology.orchestra.run.smithi196:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-22T03:07:23.867 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-09-22T03:07:23.867 DEBUG:teuthology.orchestra.run.smithi196:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-22T03:07:23.923 INFO:tasks.ceph:Setting up mon nodes... 2024-09-22T03:07:23.923 INFO:tasks.ceph:Setting up mgr nodes... 2024-09-22T03:07:23.924 DEBUG:teuthology.orchestra.run.smithi196:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-09-22T03:07:24.042 INFO:teuthology.orchestra.run.smithi196.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-09-22T03:07:24.045 INFO:tasks.ceph:Setting up mds nodes... 2024-09-22T03:07:24.045 INFO:tasks.ceph_client:Setting up client nodes... 2024-09-22T03:07:24.045 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-09-22T03:07:24.092 INFO:teuthology.orchestra.run.smithi153.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-09-22T03:07:24.105 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-09-22T03:07:24.151 INFO:teuthology.orchestra.run.smithi196.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-09-22T03:07:24.164 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-09-22T03:07:24.164 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi153.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2'}, Remote(name='ubuntu@smithi196.front.sepia.ceph.com'): {'osd.2': '/dev/vg_nvme/lv_1', 'osd.3': '/dev/vg_nvme/lv_2'}} 2024-09-22T03:07:24.164 DEBUG:teuthology.orchestra.run.smithi153:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-09-22T03:07:24.189 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2'} 2024-09-22T03:07:24.189 INFO:tasks.ceph:role: osd.0 2024-09-22T03:07:24.189 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi153.front.sepia.ceph.com 2024-09-22T03:07:24.189 DEBUG:teuthology.orchestra.run.smithi153:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-22T03:07:24.261 INFO:teuthology.orchestra.run.smithi153.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-22T03:07:24.261 INFO:teuthology.orchestra.run.smithi153.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-22T03:07:24.261 INFO:teuthology.orchestra.run.smithi153.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-22T03:07:24.261 INFO:teuthology.orchestra.run.smithi153.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-22T03:07:24.262 INFO:teuthology.orchestra.run.smithi153.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-22T03:07:24.262 INFO:teuthology.orchestra.run.smithi153.stdout: = sunit=0 swidth=0 blks 2024-09-22T03:07:24.262 INFO:teuthology.orchestra.run.smithi153.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-22T03:07:24.262 INFO:teuthology.orchestra.run.smithi153.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-22T03:07:24.262 INFO:teuthology.orchestra.run.smithi153.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-22T03:07:24.262 INFO:teuthology.orchestra.run.smithi153.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-22T03:07:24.781 INFO:teuthology.orchestra.run.smithi153.stdout:Discarding blocks...Done. 2024-09-22T03:07:24.783 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi153.front.sepia.ceph.com -o noatime 2024-09-22T03:07:24.783 DEBUG:teuthology.orchestra.run.smithi153:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-09-22T03:07:24.907 DEBUG:teuthology.orchestra.run.smithi153:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-09-22T03:07:24.939 DEBUG:teuthology.orchestra.run.smithi153:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-09-22T03:07:25.005 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2'} 2024-09-22T03:07:25.005 INFO:tasks.ceph:role: osd.1 2024-09-22T03:07:25.005 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi153.front.sepia.ceph.com 2024-09-22T03:07:25.005 DEBUG:teuthology.orchestra.run.smithi153:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-22T03:07:25.077 INFO:teuthology.orchestra.run.smithi153.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-22T03:07:25.078 INFO:teuthology.orchestra.run.smithi153.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-22T03:07:25.078 INFO:teuthology.orchestra.run.smithi153.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-22T03:07:25.078 INFO:teuthology.orchestra.run.smithi153.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-22T03:07:25.078 INFO:teuthology.orchestra.run.smithi153.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-22T03:07:25.078 INFO:teuthology.orchestra.run.smithi153.stdout: = sunit=0 swidth=0 blks 2024-09-22T03:07:25.078 INFO:teuthology.orchestra.run.smithi153.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-22T03:07:25.078 INFO:teuthology.orchestra.run.smithi153.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-22T03:07:25.078 INFO:teuthology.orchestra.run.smithi153.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-22T03:07:25.078 INFO:teuthology.orchestra.run.smithi153.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-22T03:07:25.557 INFO:teuthology.orchestra.run.smithi153.stdout:Discarding blocks...Done. 2024-09-22T03:07:25.560 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi153.front.sepia.ceph.com -o noatime 2024-09-22T03:07:25.560 DEBUG:teuthology.orchestra.run.smithi153:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-09-22T03:07:25.652 DEBUG:teuthology.orchestra.run.smithi153:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-09-22T03:07:25.681 DEBUG:teuthology.orchestra.run.smithi153:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-22T03:07:26.276 INFO:teuthology.orchestra.run.smithi153.stderr:WARNING: debug mode. Not for benchmarking or production 2024-09-22T03:07:26.277 INFO:teuthology.orchestra.run.smithi153.stderr:INFO 2024-09-22 03:07:26,276 seastar - Reactor backend: linux-aio 2024-09-22T03:07:26.296 INFO:teuthology.orchestra.run.smithi153.stderr:==27754==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-09-22T03:07:26.345 INFO:teuthology.orchestra.run.smithi153.stderr:INFO 2024-09-22 03:07:26,344 [shard 0:main] osd - bypassing the config fetch due to --no-mon-config 2024-09-22T03:07:26.346 INFO:teuthology.orchestra.run.smithi153.stderr:INFO 2024-09-22 03:07:26,344 [shard 0:main] osd - get_early_config: set --thread-affinity 1 --cpuset 0-2 2024-09-22T03:07:26.346 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG 2024-09-22 03:07:26,345 [shard 0:main] seastar - reactor::drain 2024-09-22T03:07:26.845 INFO:teuthology.orchestra.run.smithi153.stderr:----------------------------------------------------- 2024-09-22T03:07:26.845 INFO:teuthology.orchestra.run.smithi153.stderr:Suppressions used: 2024-09-22T03:07:26.845 INFO:teuthology.orchestra.run.smithi153.stderr: count bytes template 2024-09-22T03:07:26.846 INFO:teuthology.orchestra.run.smithi153.stderr: 1 8 MallocExtension::Register 2024-09-22T03:07:26.846 INFO:teuthology.orchestra.run.smithi153.stderr:----------------------------------------------------- 2024-09-22T03:07:26.846 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-22T03:07:26.900 INFO:teuthology.orchestra.run.smithi153.stderr:WARNING: debug mode. Not for benchmarking or production 2024-09-22T03:07:26.902 INFO:teuthology.orchestra.run.smithi153.stderr:INFO 2024-09-22 03:07:26,901 seastar - Reactor backend: linux-aio 2024-09-22T03:07:26.938 INFO:teuthology.orchestra.run.smithi153.stderr:==27750==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-09-22T03:07:32.004 INFO:teuthology.orchestra.run.smithi153.stderr:----------------------------------------------------- 2024-09-22T03:07:32.005 INFO:teuthology.orchestra.run.smithi153.stderr:Suppressions used: 2024-09-22T03:07:32.005 INFO:teuthology.orchestra.run.smithi153.stderr: count bytes template 2024-09-22T03:07:32.005 INFO:teuthology.orchestra.run.smithi153.stderr: 1 8 MallocExtension::Register 2024-09-22T03:07:32.005 INFO:teuthology.orchestra.run.smithi153.stderr:----------------------------------------------------- 2024-09-22T03:07:32.005 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-22T03:07:32.005 INFO:teuthology.orchestra.run.smithi153.stdout:created object store /var/lib/ceph/osd/ceph-0 for osd.0 fsid f17abef2-bc08-46d2-abf1-d47a95f0e2ee 2024-09-22T03:07:32.043 DEBUG:teuthology.orchestra.run.smithi153:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-22T03:07:32.067 DEBUG:teuthology.orchestra.run.smithi153:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-22T03:07:32.611 INFO:teuthology.orchestra.run.smithi153.stderr:WARNING: debug mode. Not for benchmarking or production 2024-09-22T03:07:32.612 INFO:teuthology.orchestra.run.smithi153.stderr:INFO 2024-09-22 03:07:32,611 seastar - Reactor backend: linux-aio 2024-09-22T03:07:32.639 INFO:teuthology.orchestra.run.smithi153.stderr:==27941==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-09-22T03:07:32.684 INFO:teuthology.orchestra.run.smithi153.stderr:INFO 2024-09-22 03:07:32,683 [shard 0:main] osd - bypassing the config fetch due to --no-mon-config 2024-09-22T03:07:32.684 INFO:teuthology.orchestra.run.smithi153.stderr:INFO 2024-09-22 03:07:32,683 [shard 0:main] osd - get_early_config: set --thread-affinity 1 --cpuset 3-5 2024-09-22T03:07:32.684 INFO:teuthology.orchestra.run.smithi153.stderr:DEBUG 2024-09-22 03:07:32,684 [shard 0:main] seastar - reactor::drain 2024-09-22T03:07:33.162 INFO:teuthology.orchestra.run.smithi153.stderr:----------------------------------------------------- 2024-09-22T03:07:33.162 INFO:teuthology.orchestra.run.smithi153.stderr:Suppressions used: 2024-09-22T03:07:33.162 INFO:teuthology.orchestra.run.smithi153.stderr: count bytes template 2024-09-22T03:07:33.162 INFO:teuthology.orchestra.run.smithi153.stderr: 1 8 MallocExtension::Register 2024-09-22T03:07:33.162 INFO:teuthology.orchestra.run.smithi153.stderr:----------------------------------------------------- 2024-09-22T03:07:33.162 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-22T03:07:33.217 INFO:teuthology.orchestra.run.smithi153.stderr:WARNING: debug mode. Not for benchmarking or production 2024-09-22T03:07:33.219 INFO:teuthology.orchestra.run.smithi153.stderr:INFO 2024-09-22 03:07:33,218 seastar - Reactor backend: linux-aio 2024-09-22T03:07:33.246 INFO:teuthology.orchestra.run.smithi153.stderr:==27939==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-09-22T03:07:38.355 INFO:teuthology.orchestra.run.smithi153.stderr:----------------------------------------------------- 2024-09-22T03:07:38.355 INFO:teuthology.orchestra.run.smithi153.stderr:Suppressions used: 2024-09-22T03:07:38.355 INFO:teuthology.orchestra.run.smithi153.stderr: count bytes template 2024-09-22T03:07:38.355 INFO:teuthology.orchestra.run.smithi153.stderr: 1 8 MallocExtension::Register 2024-09-22T03:07:38.355 INFO:teuthology.orchestra.run.smithi153.stderr:----------------------------------------------------- 2024-09-22T03:07:38.355 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-22T03:07:38.355 INFO:teuthology.orchestra.run.smithi153.stdout:created object store /var/lib/ceph/osd/ceph-1 for osd.1 fsid f17abef2-bc08-46d2-abf1-d47a95f0e2ee 2024-09-22T03:07:38.394 DEBUG:teuthology.orchestra.run.smithi153:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-22T03:07:38.419 DEBUG:teuthology.orchestra.run.smithi196:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-09-22T03:07:38.449 INFO:tasks.ceph:roles_to_devs: {'osd.2': '/dev/vg_nvme/lv_1', 'osd.3': '/dev/vg_nvme/lv_2'} 2024-09-22T03:07:38.450 INFO:tasks.ceph:role: osd.2 2024-09-22T03:07:38.450 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi196.front.sepia.ceph.com 2024-09-22T03:07:38.450 DEBUG:teuthology.orchestra.run.smithi196:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-22T03:07:38.526 INFO:teuthology.orchestra.run.smithi196.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-22T03:07:38.526 INFO:teuthology.orchestra.run.smithi196.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-22T03:07:38.526 INFO:teuthology.orchestra.run.smithi196.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-22T03:07:38.526 INFO:teuthology.orchestra.run.smithi196.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-22T03:07:38.527 INFO:teuthology.orchestra.run.smithi196.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-22T03:07:38.527 INFO:teuthology.orchestra.run.smithi196.stdout: = sunit=0 swidth=0 blks 2024-09-22T03:07:38.527 INFO:teuthology.orchestra.run.smithi196.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-22T03:07:38.527 INFO:teuthology.orchestra.run.smithi196.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-22T03:07:38.527 INFO:teuthology.orchestra.run.smithi196.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-22T03:07:38.527 INFO:teuthology.orchestra.run.smithi196.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-22T03:07:39.038 INFO:teuthology.orchestra.run.smithi196.stdout:Discarding blocks...Done. 2024-09-22T03:07:39.041 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi196.front.sepia.ceph.com -o noatime 2024-09-22T03:07:39.041 DEBUG:teuthology.orchestra.run.smithi196:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-2 2024-09-22T03:07:39.154 DEBUG:teuthology.orchestra.run.smithi196:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-09-22T03:07:39.190 DEBUG:teuthology.orchestra.run.smithi196:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-09-22T03:07:39.258 INFO:tasks.ceph:roles_to_devs: {'osd.2': '/dev/vg_nvme/lv_1', 'osd.3': '/dev/vg_nvme/lv_2'} 2024-09-22T03:07:39.258 INFO:tasks.ceph:role: osd.3 2024-09-22T03:07:39.258 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi196.front.sepia.ceph.com 2024-09-22T03:07:39.259 DEBUG:teuthology.orchestra.run.smithi196:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-22T03:07:39.332 INFO:teuthology.orchestra.run.smithi196.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-22T03:07:39.332 INFO:teuthology.orchestra.run.smithi196.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-22T03:07:39.333 INFO:teuthology.orchestra.run.smithi196.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-22T03:07:39.333 INFO:teuthology.orchestra.run.smithi196.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-22T03:07:39.333 INFO:teuthology.orchestra.run.smithi196.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-22T03:07:39.333 INFO:teuthology.orchestra.run.smithi196.stdout: = sunit=0 swidth=0 blks 2024-09-22T03:07:39.333 INFO:teuthology.orchestra.run.smithi196.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-22T03:07:39.333 INFO:teuthology.orchestra.run.smithi196.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-22T03:07:39.333 INFO:teuthology.orchestra.run.smithi196.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-22T03:07:39.333 INFO:teuthology.orchestra.run.smithi196.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-22T03:07:39.796 INFO:teuthology.orchestra.run.smithi196.stdout:Discarding blocks...Done. 2024-09-22T03:07:39.799 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi196.front.sepia.ceph.com -o noatime 2024-09-22T03:07:39.799 DEBUG:teuthology.orchestra.run.smithi196:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-3 2024-09-22T03:07:39.890 DEBUG:teuthology.orchestra.run.smithi196:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-09-22T03:07:39.920 DEBUG:teuthology.orchestra.run.smithi196:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-22T03:07:40.508 INFO:teuthology.orchestra.run.smithi196.stderr:WARNING: debug mode. Not for benchmarking or production 2024-09-22T03:07:40.509 INFO:teuthology.orchestra.run.smithi196.stderr:INFO 2024-09-22 03:07:40,509 seastar - Reactor backend: linux-aio 2024-09-22T03:07:40.538 INFO:teuthology.orchestra.run.smithi196.stderr:==27600==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-09-22T03:07:40.586 INFO:teuthology.orchestra.run.smithi196.stderr:INFO 2024-09-22 03:07:40,585 [shard 0:main] osd - bypassing the config fetch due to --no-mon-config 2024-09-22T03:07:40.587 INFO:teuthology.orchestra.run.smithi196.stderr:INFO 2024-09-22 03:07:40,585 [shard 0:main] osd - get_early_config: set --thread-affinity 1 --cpuset 0-2 2024-09-22T03:07:40.587 INFO:teuthology.orchestra.run.smithi196.stderr:DEBUG 2024-09-22 03:07:40,586 [shard 0:main] seastar - reactor::drain 2024-09-22T03:07:41.093 INFO:teuthology.orchestra.run.smithi196.stderr:----------------------------------------------------- 2024-09-22T03:07:41.093 INFO:teuthology.orchestra.run.smithi196.stderr:Suppressions used: 2024-09-22T03:07:41.093 INFO:teuthology.orchestra.run.smithi196.stderr: count bytes template 2024-09-22T03:07:41.093 INFO:teuthology.orchestra.run.smithi196.stderr: 1 8 MallocExtension::Register 2024-09-22T03:07:41.093 INFO:teuthology.orchestra.run.smithi196.stderr:----------------------------------------------------- 2024-09-22T03:07:41.094 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-09-22T03:07:41.147 INFO:teuthology.orchestra.run.smithi196.stderr:WARNING: debug mode. Not for benchmarking or production 2024-09-22T03:07:41.150 INFO:teuthology.orchestra.run.smithi196.stderr:INFO 2024-09-22 03:07:41,148 seastar - Reactor backend: linux-aio 2024-09-22T03:07:41.177 INFO:teuthology.orchestra.run.smithi196.stderr:==27596==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-09-22T03:07:46.256 INFO:teuthology.orchestra.run.smithi196.stderr:----------------------------------------------------- 2024-09-22T03:07:46.257 INFO:teuthology.orchestra.run.smithi196.stderr:Suppressions used: 2024-09-22T03:07:46.257 INFO:teuthology.orchestra.run.smithi196.stderr: count bytes template 2024-09-22T03:07:46.257 INFO:teuthology.orchestra.run.smithi196.stderr: 1 8 MallocExtension::Register 2024-09-22T03:07:46.257 INFO:teuthology.orchestra.run.smithi196.stderr:----------------------------------------------------- 2024-09-22T03:07:46.257 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-09-22T03:07:46.257 INFO:teuthology.orchestra.run.smithi196.stdout:created object store /var/lib/ceph/osd/ceph-2 for osd.2 fsid f17abef2-bc08-46d2-abf1-d47a95f0e2ee 2024-09-22T03:07:46.296 DEBUG:teuthology.orchestra.run.smithi196:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-09-22T03:07:46.321 DEBUG:teuthology.orchestra.run.smithi196:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-22T03:07:46.874 INFO:teuthology.orchestra.run.smithi196.stderr:WARNING: debug mode. Not for benchmarking or production 2024-09-22T03:07:46.874 INFO:teuthology.orchestra.run.smithi196.stderr:INFO 2024-09-22 03:07:46,873 seastar - Reactor backend: linux-aio 2024-09-22T03:07:46.897 INFO:teuthology.orchestra.run.smithi196.stderr:==27787==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-09-22T03:07:46.942 INFO:teuthology.orchestra.run.smithi196.stderr:INFO 2024-09-22 03:07:46,941 [shard 0:main] osd - bypassing the config fetch due to --no-mon-config 2024-09-22T03:07:46.943 INFO:teuthology.orchestra.run.smithi196.stderr:INFO 2024-09-22 03:07:46,941 [shard 0:main] osd - get_early_config: set --thread-affinity 1 --cpuset 3-5 2024-09-22T03:07:46.943 INFO:teuthology.orchestra.run.smithi196.stderr:DEBUG 2024-09-22 03:07:46,942 [shard 0:main] seastar - reactor::drain 2024-09-22T03:07:47.437 INFO:teuthology.orchestra.run.smithi196.stderr:----------------------------------------------------- 2024-09-22T03:07:47.437 INFO:teuthology.orchestra.run.smithi196.stderr:Suppressions used: 2024-09-22T03:07:47.437 INFO:teuthology.orchestra.run.smithi196.stderr: count bytes template 2024-09-22T03:07:47.437 INFO:teuthology.orchestra.run.smithi196.stderr: 1 8 MallocExtension::Register 2024-09-22T03:07:47.437 INFO:teuthology.orchestra.run.smithi196.stderr:----------------------------------------------------- 2024-09-22T03:07:47.437 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-09-22T03:07:47.497 INFO:teuthology.orchestra.run.smithi196.stderr:WARNING: debug mode. Not for benchmarking or production 2024-09-22T03:07:47.499 INFO:teuthology.orchestra.run.smithi196.stderr:INFO 2024-09-22 03:07:47,498 seastar - Reactor backend: linux-aio 2024-09-22T03:07:47.535 INFO:teuthology.orchestra.run.smithi196.stderr:==27785==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-09-22T03:07:52.629 INFO:teuthology.orchestra.run.smithi196.stderr:----------------------------------------------------- 2024-09-22T03:07:52.630 INFO:teuthology.orchestra.run.smithi196.stderr:Suppressions used: 2024-09-22T03:07:52.630 INFO:teuthology.orchestra.run.smithi196.stderr: count bytes template 2024-09-22T03:07:52.630 INFO:teuthology.orchestra.run.smithi196.stderr: 1 8 MallocExtension::Register 2024-09-22T03:07:52.630 INFO:teuthology.orchestra.run.smithi196.stderr:----------------------------------------------------- 2024-09-22T03:07:52.630 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-09-22T03:07:52.630 INFO:teuthology.orchestra.run.smithi196.stdout:created object store /var/lib/ceph/osd/ceph-3 for osd.3 fsid f17abef2-bc08-46d2-abf1-d47a95f0e2ee 2024-09-22T03:07:52.669 DEBUG:teuthology.orchestra.run.smithi196:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-09-22T03:07:52.693 INFO:tasks.ceph:Reading keys from all nodes... 2024-09-22T03:07:52.693 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-09-22T03:07:52.694 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-09-22T03:07:52.724 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-09-22T03:07:52.725 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-09-22T03:07:52.794 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-09-22T03:07:52.794 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-09-22T03:07:52.819 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-09-22T03:07:52.819 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-09-22T03:07:52.883 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-09-22T03:07:52.883 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-09-22T03:07:52.948 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-09-22T03:07:52.948 DEBUG:teuthology.orchestra.run.smithi153:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-09-22T03:07:52.967 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-09-22T03:07:52.967 DEBUG:teuthology.orchestra.run.smithi196:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-09-22T03:07:53.004 INFO:tasks.ceph:Adding keys to all mons... 2024-09-22T03:07:53.004 DEBUG:teuthology.orchestra.run.smithi153:> sudo tee -a /etc/ceph/ceph.keyring 2024-09-22T03:07:53.035 INFO:teuthology.orchestra.run.smithi153.stdout:[osd.0] 2024-09-22T03:07:53.035 INFO:teuthology.orchestra.run.smithi153.stdout: key = AQDuie9m9spiOxAADeUWGUj6zKbe0Th7X3DMLg== 2024-09-22T03:07:53.036 INFO:teuthology.orchestra.run.smithi153.stdout:[osd.1] 2024-09-22T03:07:53.036 INFO:teuthology.orchestra.run.smithi153.stdout: key = AQD1ie9m5SsaEhAADwj3E1ocFLS1khDFtZGXcA== 2024-09-22T03:07:53.036 INFO:teuthology.orchestra.run.smithi153.stdout:[mgr.x] 2024-09-22T03:07:53.036 INFO:teuthology.orchestra.run.smithi153.stdout: key = AQDsie9mrjJ6AhAA6VlIckmG/hFw8trwsbeG/g== 2024-09-22T03:07:53.036 INFO:teuthology.orchestra.run.smithi153.stdout:[osd.2] 2024-09-22T03:07:53.036 INFO:teuthology.orchestra.run.smithi153.stdout: key = AQD9ie9mls7xDRAA8HEl29AcRZgOl9LkGQg5cg== 2024-09-22T03:07:53.036 INFO:teuthology.orchestra.run.smithi153.stdout:[osd.3] 2024-09-22T03:07:53.036 INFO:teuthology.orchestra.run.smithi153.stdout: key = AQADiu9myIJOIxAAwFk2Yzv7bd75TaUyuFwAiQ== 2024-09-22T03:07:53.036 INFO:teuthology.orchestra.run.smithi153.stdout:[client.0] 2024-09-22T03:07:53.036 INFO:teuthology.orchestra.run.smithi153.stdout: key = AQDsie9m1LJ0BRAAtffEyuj5sC0IUy60bB9WdQ== 2024-09-22T03:07:53.036 INFO:teuthology.orchestra.run.smithi153.stdout:[client.1] 2024-09-22T03:07:53.036 INFO:teuthology.orchestra.run.smithi153.stdout: key = AQDsie9mwzD+CBAA036oaN9z+aqbe5FHdg26WA== 2024-09-22T03:07:53.037 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-22T03:07:53.144 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-22T03:07:53.203 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-22T03:07:53.259 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-22T03:07:53.316 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-22T03:07:53.373 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-22T03:07:53.433 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-22T03:07:53.492 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-09-22T03:07:53.492 DEBUG:teuthology.orchestra.run.smithi153:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-09-22T03:07:53.517 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-22T03:07:53.620 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:07:53.618+0000 7f005c231b40 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:07:53.620 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:07:53.618+0000 7f005c231b40 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:07:53.633 DEBUG:teuthology.orchestra.run.smithi153:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-09-22T03:07:53.658 DEBUG:teuthology.orchestra.run.smithi153:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-09-22T03:07:53.712 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-09-22T03:07:53.713 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-22T03:07:53.713 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-22T03:07:53.755 INFO:tasks.ceph.mon.a:Started 2024-09-22T03:07:53.755 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-09-22T03:07:53.755 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-09-22T03:07:53.755 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-09-22T03:07:53.758 INFO:tasks.ceph.mgr.x:Started 2024-09-22T03:07:53.758 INFO:tasks.ceph:Setting crush tunables to default 2024-09-22T03:07:53.758 DEBUG:teuthology.orchestra.run.smithi153:> sudo ceph --cluster ceph osd crush tunables default 2024-09-22T03:07:53.833 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-22T03:07:53.830+0000 7f6204cdbb40 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:07:53.834 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-22T03:07:53.830+0000 7f6204cdbb40 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:07:53.834 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-22T03:07:53.830+0000 7f6204cdbb40 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:07:53.846 INFO:tasks.ceph.mgr.x.smithi196.stderr:2024-09-22T03:07:53.845+0000 7fdfb7541f00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:07:53.865 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:07:53.857+0000 7ff4ffe4f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:07:53.871 INFO:tasks.ceph.mgr.x.smithi196.stderr:2024-09-22T03:07:53.869+0000 7fdfb7541f00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:07:53.871 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:07:53.869+0000 7ff4ffe4f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:07:53.871 INFO:tasks.ceph.mgr.x.smithi196.stderr:2024-09-22T03:07:53.869+0000 7fdfb7541f00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:07:53.938 INFO:teuthology.orchestra.run.smithi153.stderr:adjusted tunables profile to default 2024-09-22T03:07:53.949 INFO:tasks.ceph:check_enable_crimson: True 2024-09-22T03:07:53.949 INFO:tasks.ceph:check_enable_crimson: setting set-allow-crimson 2024-09-22T03:07:53.949 DEBUG:teuthology.orchestra.run.smithi153:> sudo ceph --cluster ceph osd set-allow-crimson --yes-i-really-mean-it 2024-09-22T03:07:53.972 INFO:tasks.ceph.mgr.x.smithi196.stderr:2024-09-22T03:07:53.971+0000 7fdfb7541f00 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-22T03:07:54.013 INFO:tasks.ceph.mgr.x.smithi196.stderr:2024-09-22T03:07:54.011+0000 7fdfb7541f00 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-22T03:07:54.050 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:07:54.044+0000 7f29f2a63640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:07:54.053 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:07:54.052+0000 7f29f2a63640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:07:54.128 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-22T03:07:54.127+0000 7f61fcbf1640 -1 WARNING: experimental feature 'crimson' is enabled 2024-09-22T03:07:54.129 INFO:tasks.ceph.mon.a.smithi153.stderr:Please be aware that this feature is experimental, untested, 2024-09-22T03:07:54.129 INFO:tasks.ceph.mon.a.smithi153.stderr:unsupported, and may result in data corruption, data loss, 2024-09-22T03:07:54.129 INFO:tasks.ceph.mon.a.smithi153.stderr:and/or irreparable damage to your cluster. Do not use 2024-09-22T03:07:54.129 INFO:tasks.ceph.mon.a.smithi153.stderr:feature with important data. 2024-09-22T03:07:54.129 INFO:tasks.ceph.mon.a.smithi153.stderr: 2024-09-22T03:07:54.147 INFO:tasks.ceph.mgr.x.smithi196.stderr:2024-09-22T03:07:54.145+0000 7fdfb7541f00 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-22T03:07:54.197 INFO:tasks.ceph.mgr.x.smithi196.stderr:2024-09-22T03:07:54.195+0000 7fdfb7541f00 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-22T03:07:54.359 INFO:tasks.ceph.mgr.x.smithi196.stderr:2024-09-22T03:07:54.357+0000 7fdfb7541f00 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-22T03:07:54.468 INFO:tasks.ceph.mgr.x.smithi196.stderr:2024-09-22T03:07:54.466+0000 7fdfb7541f00 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-22T03:07:54.808 INFO:tasks.ceph.mgr.x.smithi196.stderr:2024-09-22T03:07:54.806+0000 7fdfb7541f00 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-22T03:07:54.861 INFO:tasks.ceph.mgr.x.smithi196.stderr:2024-09-22T03:07:54.860+0000 7fdfb7541f00 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-22T03:07:54.952 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-09-22T03:07:54.952 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-09-22T03:07:54.953 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-09-22T03:07:54.980 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-09-22T03:07:54.980 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-09-22T03:07:55.011 INFO:tasks.ceph.mgr.x.smithi196.stderr:2024-09-22T03:07:55.009+0000 7fdfb7541f00 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-22T03:07:55.047 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-09-22T03:07:55.047 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-09-22T03:07:55.072 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-09-22T03:07:55.072 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-09-22T03:07:55.138 DEBUG:teuthology.orchestra.run.smithi196:> sudo ceph --cluster ceph osd new 35bc7c89-cb7c-4664-bed6-4088394939d9 0 2024-09-22T03:07:55.283 INFO:teuthology.orchestra.run.smithi196.stderr:2024-09-22T03:07:55.276+0000 7f7228c6c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:07:55.285 INFO:teuthology.orchestra.run.smithi196.stderr:2024-09-22T03:07:55.284+0000 7f7228c6c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:07:55.362 INFO:teuthology.orchestra.run.smithi196.stdout:0 2024-09-22T03:07:55.373 DEBUG:teuthology.orchestra.run.smithi196:> sudo ceph --cluster ceph osd new 513ab51d-4bac-4f1a-ade1-755e26e4c8e9 1 2024-09-22T03:07:55.476 INFO:teuthology.orchestra.run.smithi196.stderr:2024-09-22T03:07:55.469+0000 7fa200b97640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:07:55.479 INFO:teuthology.orchestra.run.smithi196.stderr:2024-09-22T03:07:55.477+0000 7fa200b97640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:07:55.556 INFO:teuthology.orchestra.run.smithi196.stdout:1 2024-09-22T03:07:55.567 DEBUG:teuthology.orchestra.run.smithi196:> sudo ceph --cluster ceph osd new dd57bc3a-b76e-4c7f-8935-6c2b7eb8ef7b 2 2024-09-22T03:07:55.671 INFO:teuthology.orchestra.run.smithi196.stderr:2024-09-22T03:07:55.664+0000 7fb1bff97640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:07:55.674 INFO:teuthology.orchestra.run.smithi196.stderr:2024-09-22T03:07:55.672+0000 7fb1bff97640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:07:55.750 INFO:teuthology.orchestra.run.smithi196.stdout:2 2024-09-22T03:07:55.762 DEBUG:teuthology.orchestra.run.smithi196:> sudo ceph --cluster ceph osd new 76e7924f-5812-47ee-a5ee-ec837d12f327 3 2024-09-22T03:07:55.867 INFO:teuthology.orchestra.run.smithi196.stderr:2024-09-22T03:07:55.860+0000 7fe79ed97640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:07:55.869 INFO:teuthology.orchestra.run.smithi196.stderr:2024-09-22T03:07:55.868+0000 7fe79ed97640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:07:55.946 INFO:teuthology.orchestra.run.smithi196.stdout:3 2024-09-22T03:07:55.957 DEBUG:teuthology.orchestra.run.smithi153:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.0.log 2024-09-22T03:07:55.992 INFO:tasks.ceph.osd.0:Restarting daemon 2024-09-22T03:07:55.992 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2>> /var/log/ceph/ceph-osd.0.log 2024-09-22T03:07:56.019 INFO:tasks.ceph.mgr.x.smithi196.stderr:2024-09-22T03:07:56.017+0000 7fdfb7541f00 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-22T03:07:56.035 INFO:tasks.ceph.osd.0:Started 2024-09-22T03:07:56.035 DEBUG:teuthology.orchestra.run.smithi153:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.1.log 2024-09-22T03:07:56.068 INFO:tasks.ceph.osd.1:Restarting daemon 2024-09-22T03:07:56.068 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2>> /var/log/ceph/ceph-osd.1.log 2024-09-22T03:07:56.071 INFO:tasks.ceph.mgr.x.smithi196.stderr:2024-09-22T03:07:56.069+0000 7fdfb7541f00 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-22T03:07:56.111 INFO:tasks.ceph.osd.1:Started 2024-09-22T03:07:56.111 DEBUG:teuthology.orchestra.run.smithi196:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.2.log 2024-09-22T03:07:56.140 INFO:tasks.ceph.osd.2:Restarting daemon 2024-09-22T03:07:56.140 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2>> /var/log/ceph/ceph-osd.2.log 2024-09-22T03:07:56.183 INFO:tasks.ceph.osd.2:Started 2024-09-22T03:07:56.183 DEBUG:teuthology.orchestra.run.smithi196:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.3.log 2024-09-22T03:07:56.212 INFO:tasks.ceph.osd.3:Restarting daemon 2024-09-22T03:07:56.212 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2>> /var/log/ceph/ceph-osd.3.log 2024-09-22T03:07:56.254 INFO:tasks.ceph.osd.3:Started 2024-09-22T03:07:56.255 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-22T03:07:56.378 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:07:56.372+0000 7fd5be4ba640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:07:56.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:07:56.380+0000 7fd5be4ba640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:07:56.450 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-22T03:07:56.450 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":7,"fsid":"f17abef2-bc08-46d2-abf1-d47a95f0e2ee","created":"2024-09-22T03:07:53.859391+0000","modified":"2024-09-22T03:07:55.944112+0000","last_up_change":"0.000000","last_in_change":"2024-09-22T03:07:55.944112+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"35bc7c89-cb7c-4664-bed6-4088394939d9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"513ab51d-4bac-4f1a-ade1-755e26e4c8e9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dd57bc3a-b76e-4c7f-8935-6c2b7eb8ef7b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"76e7924f-5812-47ee-a5ee-ec837d12f327","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-22T03:07:56.464 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-09-22T03:07:56.464 INFO:tasks.ceph:Waiting for OSDs to come up 2024-09-22T03:07:56.496 INFO:tasks.ceph.mgr.x.smithi196.stderr:2024-09-22T03:07:56.494+0000 7fdfb7541f00 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-22T03:07:56.600 INFO:tasks.ceph.mgr.x.smithi196.stderr:2024-09-22T03:07:56.599+0000 7fdfb7541f00 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-22T03:07:56.722 INFO:tasks.ceph.mgr.x.smithi196.stderr:2024-09-22T03:07:56.720+0000 7fdfb7541f00 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-22T03:07:56.772 INFO:tasks.ceph.mgr.x.smithi196.stderr:2024-09-22T03:07:56.771+0000 7fdfb7541f00 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-22T03:07:56.867 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-22T03:07:56.891 INFO:tasks.ceph.mgr.x.smithi196.stderr:2024-09-22T03:07:56.889+0000 7fdfb7541f00 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-22T03:07:56.947 INFO:tasks.ceph.mgr.x.smithi196.stderr:2024-09-22T03:07:56.945+0000 7fdfb7541f00 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-22T03:07:56.969 INFO:teuthology.misc.health.smithi153.stderr:2024-09-22T03:07:56.964+0000 7f4db0397640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:07:56.972 INFO:teuthology.misc.health.smithi153.stderr:2024-09-22T03:07:56.971+0000 7f4db0397640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:07:56.995 INFO:tasks.ceph.mgr.x.smithi196.stderr:2024-09-22T03:07:56.993+0000 7fdfb7541f00 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-22T03:07:57.037 INFO:teuthology.misc.health.smithi153.stdout: 2024-09-22T03:07:57.037 INFO:teuthology.misc.health.smithi153.stdout:{"epoch":7,"fsid":"f17abef2-bc08-46d2-abf1-d47a95f0e2ee","created":"2024-09-22T03:07:53.859391+0000","modified":"2024-09-22T03:07:55.944112+0000","last_up_change":"0.000000","last_in_change":"2024-09-22T03:07:55.944112+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"35bc7c89-cb7c-4664-bed6-4088394939d9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"513ab51d-4bac-4f1a-ade1-755e26e4c8e9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dd57bc3a-b76e-4c7f-8935-6c2b7eb8ef7b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"76e7924f-5812-47ee-a5ee-ec837d12f327","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-22T03:07:57.047 DEBUG:teuthology.misc:0 of 4 OSDs are up 2024-09-22T03:07:57.117 INFO:tasks.ceph.mgr.x.smithi196.stderr:2024-09-22T03:07:57.116+0000 7fdfb7541f00 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-22T03:07:57.172 INFO:tasks.ceph.mgr.x.smithi196.stderr:2024-09-22T03:07:57.170+0000 7fdfb7541f00 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-22T03:07:57.228 INFO:tasks.ceph.mgr.x.smithi196.stderr:2024-09-22T03:07:57.226+0000 7fdfb7541f00 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-22T03:07:57.302 INFO:tasks.ceph.mgr.x.smithi196.stderr:2024-09-22T03:07:57.301+0000 7fdfb7541f00 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-22T03:07:57.351 INFO:tasks.ceph.mgr.x.smithi196.stderr:2024-09-22T03:07:57.349+0000 7fdfb7541f00 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-22T03:07:57.399 INFO:tasks.ceph.mgr.x.smithi196.stderr:2024-09-22T03:07:57.397+0000 7fdfb7541f00 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-22T03:07:58.097 INFO:tasks.ceph.mgr.x.smithi196.stderr:2024-09-22T03:07:58.095+0000 7fdfb7541f00 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-22T03:07:59.109 INFO:tasks.ceph.mgr.x.smithi196.stderr:2024-09-22T03:07:59.107+0000 7fdf566ea640 -1 mgr.server handle_report got status from non-daemon mon.a 2024-09-22T03:08:03.451 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-22T03:08:03.559 INFO:teuthology.misc.health.smithi153.stderr:2024-09-22T03:08:03.553+0000 7febcb557640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:08:03.562 INFO:teuthology.misc.health.smithi153.stderr:2024-09-22T03:08:03.560+0000 7febcb557640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:08:03.738 INFO:teuthology.misc.health.smithi153.stdout: 2024-09-22T03:08:03.738 INFO:teuthology.misc.health.smithi153.stdout:{"epoch":10,"fsid":"f17abef2-bc08-46d2-abf1-d47a95f0e2ee","created":"2024-09-22T03:07:53.859391+0000","modified":"2024-09-22T03:08:02.112316+0000","last_up_change":"2024-09-22T03:08:02.112316+0000","last_in_change":"2024-09-22T03:07:55.944112+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"35bc7c89-cb7c-4664-bed6-4088394939d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6805","nonce":406291079}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":406291079}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":406291079}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6801","nonce":406291079}]},"public_addr":"172.21.15.153:6805/406291079","cluster_addr":"172.21.15.153:6804/406291079","heartbeat_back_addr":"172.21.15.153:6800/406291079","heartbeat_front_addr":"172.21.15.153:6801/406291079","state":["exists","up"]},{"osd":1,"uuid":"513ab51d-4bac-4f1a-ade1-755e26e4c8e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6803","nonce":1323003415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":1323003415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":1323003415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6807","nonce":1323003415}]},"public_addr":"172.21.15.153:6803/1323003415","cluster_addr":"172.21.15.153:6806/1323003415","heartbeat_back_addr":"172.21.15.153:6802/1323003415","heartbeat_front_addr":"172.21.15.153:6807/1323003415","state":["exists","up"]},{"osd":2,"uuid":"dd57bc3a-b76e-4c7f-8935-6c2b7eb8ef7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6806","nonce":527153512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6805","nonce":527153512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6809","nonce":527153512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6810","nonce":527153512}]},"public_addr":"172.21.15.196:6806/527153512","cluster_addr":"172.21.15.196:6805/527153512","heartbeat_back_addr":"172.21.15.196:6809/527153512","heartbeat_front_addr":"172.21.15.196:6810/527153512","state":["exists","up"]},{"osd":3,"uuid":"76e7924f-5812-47ee-a5ee-ec837d12f327","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6804","nonce":3375769528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6803","nonce":3375769528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6807","nonce":3375769528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6808","nonce":3375769528}]},"public_addr":"172.21.15.196:6804/3375769528","cluster_addr":"172.21.15.196:6803/3375769528","heartbeat_back_addr":"172.21.15.196:6807/3375769528","heartbeat_front_addr":"172.21.15.196:6808/3375769528","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-22T03:08:03.750 DEBUG:teuthology.misc:4 of 4 OSDs are up 2024-09-22T03:08:03.751 INFO:tasks.ceph:Creating RBD pool 2024-09-22T03:08:03.751 DEBUG:teuthology.orchestra.run.smithi153:> sudo ceph --cluster ceph osd pool create rbd 8 2024-09-22T03:08:03.855 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:08:03.846+0000 7f8a11717640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:08:03.857 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:08:03.856+0000 7f8a11717640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:08:04.137 INFO:teuthology.orchestra.run.smithi153.stderr:pool 'rbd' created 2024-09-22T03:08:04.158 DEBUG:teuthology.orchestra.run.smithi153:> rbd --cluster ceph pool init rbd 2024-09-22T03:08:04.306 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:08:04.304+0000 7f1a78866d00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:08:04.307 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:08:04.304+0000 7f1a78866d00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:08:04.310 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:08:04.309+0000 7f1a78866d00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:08:07.170 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-09-22T03:08:07.171 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-09-22T03:08:07.171 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-09-22T03:08:07.171 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-09-22T03:08:07.172 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-09-22T03:08:07.280 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:08:07.275+0000 7f045130e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:08:07.283 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:08:07.282+0000 7f045130e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:08:07.511 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-22T03:08:07.520 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":5,"flags":0,"active_gid":4099,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6800","nonce":3255235497},{"type":"v1","addr":"172.21.15.196:6801","nonce":3255235497}]},"active_addr":"172.21.15.196:6801/3255235497","active_change":"2024-09-22T03:07:58.097995+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.5","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.196:0","nonce":1770545978}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.196:0","nonce":1166286858}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.196:0","nonce":2002234321}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.196:0","nonce":230766246}]}]} 2024-09-22T03:08:07.523 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-09-22T03:08:07.523 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-09-22T03:08:07.523 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-22T03:08:07.626 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:08:07.621+0000 7f45c8b97640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:08:07.629 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:08:07.627+0000 7f45c8b97640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:08:07.806 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-22T03:08:07.806 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":14,"fsid":"f17abef2-bc08-46d2-abf1-d47a95f0e2ee","created":"2024-09-22T03:07:53.859391+0000","modified":"2024-09-22T03:08:07.141344+0000","last_up_change":"2024-09-22T03:08:02.112316+0000","last_in_change":"2024-09-22T03:07:55.944112+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2024-09-22T03:08:04.049314+0000","flags":270369,"flags_names":"hashpspool,nopgchange,selfmanaged_snaps,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":14,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".mgr","create_time":"2024-09-22T03:08:04.133626+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.75,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"35bc7c89-cb7c-4664-bed6-4088394939d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6805","nonce":406291079}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":406291079}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":406291079}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6801","nonce":406291079}]},"public_addr":"172.21.15.153:6805/406291079","cluster_addr":"172.21.15.153:6804/406291079","heartbeat_back_addr":"172.21.15.153:6800/406291079","heartbeat_front_addr":"172.21.15.153:6801/406291079","state":["exists","up"]},{"osd":1,"uuid":"513ab51d-4bac-4f1a-ade1-755e26e4c8e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6803","nonce":1323003415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":1323003415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":1323003415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6807","nonce":1323003415}]},"public_addr":"172.21.15.153:6803/1323003415","cluster_addr":"172.21.15.153:6806/1323003415","heartbeat_back_addr":"172.21.15.153:6802/1323003415","heartbeat_front_addr":"172.21.15.153:6807/1323003415","state":["exists","up"]},{"osd":2,"uuid":"dd57bc3a-b76e-4c7f-8935-6c2b7eb8ef7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6806","nonce":527153512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6805","nonce":527153512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6809","nonce":527153512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6810","nonce":527153512}]},"public_addr":"172.21.15.196:6806/527153512","cluster_addr":"172.21.15.196:6805/527153512","heartbeat_back_addr":"172.21.15.196:6809/527153512","heartbeat_front_addr":"172.21.15.196:6810/527153512","state":["exists","up"]},{"osd":3,"uuid":"76e7924f-5812-47ee-a5ee-ec837d12f327","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6804","nonce":3375769528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6803","nonce":3375769528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6807","nonce":3375769528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6808","nonce":3375769528}]},"public_addr":"172.21.15.196:6804/3375769528","cluster_addr":"172.21.15.196:6803/3375769528","heartbeat_back_addr":"172.21.15.196:6807/3375769528","heartbeat_front_addr":"172.21.15.196:6808/3375769528","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":1,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":1,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-22T03:08:07.818 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-09-22T03:08:07.819 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-22T03:08:07.922 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:08:07.917+0000 7f16f8917640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:08:07.925 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:08:07.923+0000 7f16f8917640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:08:08.103 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-22T03:08:08.103 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":14,"fsid":"f17abef2-bc08-46d2-abf1-d47a95f0e2ee","created":"2024-09-22T03:07:53.859391+0000","modified":"2024-09-22T03:08:07.141344+0000","last_up_change":"2024-09-22T03:08:02.112316+0000","last_in_change":"2024-09-22T03:07:55.944112+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2024-09-22T03:08:04.049314+0000","flags":270369,"flags_names":"hashpspool,nopgchange,selfmanaged_snaps,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":14,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".mgr","create_time":"2024-09-22T03:08:04.133626+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.75,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"35bc7c89-cb7c-4664-bed6-4088394939d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6805","nonce":406291079}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":406291079}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":406291079}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6801","nonce":406291079}]},"public_addr":"172.21.15.153:6805/406291079","cluster_addr":"172.21.15.153:6804/406291079","heartbeat_back_addr":"172.21.15.153:6800/406291079","heartbeat_front_addr":"172.21.15.153:6801/406291079","state":["exists","up"]},{"osd":1,"uuid":"513ab51d-4bac-4f1a-ade1-755e26e4c8e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6803","nonce":1323003415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":1323003415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":1323003415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6807","nonce":1323003415}]},"public_addr":"172.21.15.153:6803/1323003415","cluster_addr":"172.21.15.153:6806/1323003415","heartbeat_back_addr":"172.21.15.153:6802/1323003415","heartbeat_front_addr":"172.21.15.153:6807/1323003415","state":["exists","up"]},{"osd":2,"uuid":"dd57bc3a-b76e-4c7f-8935-6c2b7eb8ef7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6806","nonce":527153512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6805","nonce":527153512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6809","nonce":527153512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6810","nonce":527153512}]},"public_addr":"172.21.15.196:6806/527153512","cluster_addr":"172.21.15.196:6805/527153512","heartbeat_back_addr":"172.21.15.196:6809/527153512","heartbeat_front_addr":"172.21.15.196:6810/527153512","state":["exists","up"]},{"osd":3,"uuid":"76e7924f-5812-47ee-a5ee-ec837d12f327","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6804","nonce":3375769528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6803","nonce":3375769528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6807","nonce":3375769528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6808","nonce":3375769528}]},"public_addr":"172.21.15.196:6804/3375769528","cluster_addr":"172.21.15.196:6803/3375769528","heartbeat_back_addr":"172.21.15.196:6807/3375769528","heartbeat_front_addr":"172.21.15.196:6808/3375769528","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":1,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":1,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-22T03:08:08.116 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-09-22T03:08:08.117 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-09-22T03:08:08.117 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-09-22T03:08:08.118 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-09-22T03:08:08.234 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:08:08.228+0000 7facd04d7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:08:08.235 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:08:08.228+0000 7ff49be6c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:08:08.235 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:08:08.229+0000 7fad4c10e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:08:08.235 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:08:08.229+0000 7f0ce364f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:08:08.238 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:08:08.237+0000 7ff49be6c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:08:08.239 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:08:08.237+0000 7facd04d7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:08:08.239 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:08:08.237+0000 7fad4c10e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:08:08.239 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:08:08.238+0000 7f0ce364f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:08:08.268 INFO:teuthology.orchestra.run.smithi153.stdout:42949672966 2024-09-22T03:08:08.269 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-22T03:08:08.270 INFO:teuthology.orchestra.run.smithi153.stdout:38654705671 2024-09-22T03:08:08.270 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-22T03:08:08.273 INFO:teuthology.orchestra.run.smithi153.stdout:42949672966 2024-09-22T03:08:08.273 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-22T03:08:08.275 INFO:teuthology.orchestra.run.smithi153.stdout:42949672966 2024-09-22T03:08:08.275 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-22T03:08:08.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:08:08.375+0000 7f770c397640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:08:08.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:08:08.375+0000 7f0b8bb4e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:08:08.385 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:08:08.384+0000 7f770c397640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:08:08.386 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:08:08.384+0000 7f0b8bb4e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:08:08.428 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:08:08.422+0000 7fbb0a597640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:08:08.432 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:08:08.430+0000 7fbb0a597640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:08:08.435 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:08:08.428+0000 7f4cf106c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:08:08.448 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:08:08.447+0000 7f4cf106c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:08:08.589 INFO:teuthology.orchestra.run.smithi153.stdout:38654705668 2024-09-22T03:08:08.591 INFO:teuthology.orchestra.run.smithi153.stdout:42949672966 2024-09-22T03:08:08.604 INFO:tasks.ceph.ceph_manager.ceph:need seq 42949672966 got 42949672966 for osd.1 2024-09-22T03:08:08.604 DEBUG:teuthology.parallel:result is None 2024-09-22T03:08:08.607 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705671 got 38654705668 for osd.0 2024-09-22T03:08:08.671 INFO:teuthology.orchestra.run.smithi153.stdout:42949672966 2024-09-22T03:08:08.679 INFO:teuthology.orchestra.run.smithi153.stdout:42949672966 2024-09-22T03:08:08.683 INFO:tasks.ceph.ceph_manager.ceph:need seq 42949672966 got 42949672966 for osd.2 2024-09-22T03:08:08.683 DEBUG:teuthology.parallel:result is None 2024-09-22T03:08:08.692 INFO:tasks.ceph.ceph_manager.ceph:need seq 42949672966 got 42949672966 for osd.3 2024-09-22T03:08:08.692 DEBUG:teuthology.parallel:result is None 2024-09-22T03:08:09.608 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-22T03:08:09.715 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:08:09.709+0000 7f8ea044f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:08:09.719 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:08:09.717+0000 7f8ea044f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:08:09.896 INFO:teuthology.orchestra.run.smithi153.stdout:38654705668 2024-09-22T03:08:09.909 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705671 got 38654705668 for osd.0 2024-09-22T03:08:10.910 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-22T03:08:11.029 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:08:11.023+0000 7f6ae374e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:08:11.032 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:08:11.030+0000 7f6ae374e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:08:11.210 INFO:teuthology.orchestra.run.smithi153.stdout:38654705671 2024-09-22T03:08:11.222 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705671 got 38654705671 for osd.0 2024-09-22T03:08:11.222 DEBUG:teuthology.parallel:result is None 2024-09-22T03:08:11.223 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-22T03:08:11.223 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-22T03:08:11.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:08:11.322+0000 7f69d7d97640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:08:11.330 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:08:11.328+0000 7f69d7d97640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:08:11.502 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-22T03:08:11.503 INFO:teuthology.orchestra.run.smithi153.stderr:dumped all 2024-09-22T03:08:11.515 INFO:teuthology.orchestra.run.smithi153.stdout:{"pg_ready":true,"pg_map":{"version":11,"stamp":"2024-09-22T03:08:10.104771+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":139,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":27,"acting":27,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":108020,"kb_used_data":656,"kb_used_omap":0,"kb_used_meta":107264,"kb_avail":377379340,"statfs":{"total":386547056640,"available":386436444160,"internally_reserved":0,"allocated":671744,"data_stored":1894113,"data_compressed":89886,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":109838336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.965670"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":16,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-22T03:08:07.154077+0000","last_change":"2024-09-22T03:08:07.157007+0000","last_active":"2024-09-22T03:08:07.154077+0000","last_peered":"2024-09-22T03:08:07.154077+0000","last_clean":"2024-09-22T03:08:07.154077+0000","last_became_active":"2024-09-22T03:08:05.192724+0000","last_became_peered":"2024-09-22T03:08:05.192724+0000","last_unstale":"2024-09-22T03:08:07.154077+0000","last_undegraded":"2024-09-22T03:08:07.154077+0000","last_fullsized":"2024-09-22T03:08:07.154077+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":16,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-22T03:08:07.154489+0000","last_change":"2024-09-22T03:08:07.156852+0000","last_active":"2024-09-22T03:08:07.154489+0000","last_peered":"2024-09-22T03:08:07.154489+0000","last_clean":"2024-09-22T03:08:07.154489+0000","last_became_active":"2024-09-22T03:08:05.203704+0000","last_became_peered":"2024-09-22T03:08:05.203704+0000","last_unstale":"2024-09-22T03:08:07.154489+0000","last_undegraded":"2024-09-22T03:08:07.154489+0000","last_fullsized":"2024-09-22T03:08:07.154489+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":16,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-22T03:08:07.155483+0000","last_change":"2024-09-22T03:08:07.156760+0000","last_active":"2024-09-22T03:08:07.155483+0000","last_peered":"2024-09-22T03:08:07.155483+0000","last_clean":"2024-09-22T03:08:07.155483+0000","last_became_active":"2024-09-22T03:08:05.189504+0000","last_became_peered":"2024-09-22T03:08:05.189504+0000","last_unstale":"2024-09-22T03:08:07.155483+0000","last_undegraded":"2024-09-22T03:08:07.155483+0000","last_fullsized":"2024-09-22T03:08:07.155483+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":16,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-22T03:08:07.152094+0000","last_change":"2024-09-22T03:08:07.153737+0000","last_active":"2024-09-22T03:08:07.152094+0000","last_peered":"2024-09-22T03:08:07.152094+0000","last_clean":"2024-09-22T03:08:07.152094+0000","last_became_active":"2024-09-22T03:08:05.184155+0000","last_became_peered":"2024-09-22T03:08:05.184155+0000","last_unstale":"2024-09-22T03:08:07.152094+0000","last_undegraded":"2024-09-22T03:08:07.152094+0000","last_fullsized":"2024-09-22T03:08:07.152094+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"12'97","reported_seq":142,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-22T03:08:07.155728+0000","last_change":"2024-09-22T03:08:05.205288+0000","last_active":"2024-09-22T03:08:07.155728+0000","last_peered":"2024-09-22T03:08:07.155728+0000","last_clean":"2024-09-22T03:08:07.155728+0000","last_became_active":"2024-09-22T03:08:05.195537+0000","last_became_peered":"2024-09-22T03:08:05.195537+0000","last_unstale":"2024-09-22T03:08:07.155728+0000","last_undegraded":"2024-09-22T03:08:07.155728+0000","last_fullsized":"2024-09-22T03:08:07.155728+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-22T03:08:07.155274+0000","last_change":"2024-09-22T03:08:07.156930+0000","last_active":"2024-09-22T03:08:07.155274+0000","last_peered":"2024-09-22T03:08:07.155274+0000","last_clean":"2024-09-22T03:08:07.155274+0000","last_became_active":"2024-09-22T03:08:05.183326+0000","last_became_peered":"2024-09-22T03:08:05.183326+0000","last_unstale":"2024-09-22T03:08:07.155274+0000","last_undegraded":"2024-09-22T03:08:07.155274+0000","last_fullsized":"2024-09-22T03:08:07.155274+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"13'1","reported_seq":18,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-22T03:08:07.155023+0000","last_change":"2024-09-22T03:08:07.155023+0000","last_active":"2024-09-22T03:08:07.155023+0000","last_peered":"2024-09-22T03:08:07.155023+0000","last_clean":"2024-09-22T03:08:07.155023+0000","last_became_active":"2024-09-22T03:08:05.190985+0000","last_became_peered":"2024-09-22T03:08:05.190985+0000","last_unstale":"2024-09-22T03:08:07.155023+0000","last_undegraded":"2024-09-22T03:08:07.155023+0000","last_fullsized":"2024-09-22T03:08:07.155023+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":16,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-22T03:08:07.153136+0000","last_change":"2024-09-22T03:08:07.155571+0000","last_active":"2024-09-22T03:08:07.153136+0000","last_peered":"2024-09-22T03:08:07.153136+0000","last_clean":"2024-09-22T03:08:07.153136+0000","last_became_active":"2024-09-22T03:08:05.189949+0000","last_became_peered":"2024-09-22T03:08:05.189949+0000","last_unstale":"2024-09-22T03:08:07.153136+0000","last_undegraded":"2024-09-22T03:08:07.153136+0000","last_fullsized":"2024-09-22T03:08:07.153136+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":16,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-22T03:08:07.152054+0000","last_change":"2024-09-22T03:08:07.154145+0000","last_active":"2024-09-22T03:08:07.152054+0000","last_peered":"2024-09-22T03:08:07.152054+0000","last_clean":"2024-09-22T03:08:07.152054+0000","last_became_active":"2024-09-22T03:08:05.198559+0000","last_became_peered":"2024-09-22T03:08:05.198559+0000","last_unstale":"2024-09-22T03:08:07.152054+0000","last_undegraded":"2024-09-22T03:08:07.152054+0000","last_fullsized":"2024-09-22T03:08:07.152054+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29962,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":1},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":24,"acting":24,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":10,"seq":42949672966,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26952,"kb_used_data":120,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344888,"statfs":{"total":96636764160,"available":96609165312,"internally_reserved":0,"allocated":122880,"data_stored":30757,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":10,"seq":42949672966,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27024,"kb_used_data":180,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344816,"statfs":{"total":96636764160,"available":96609091584,"internally_reserved":0,"allocated":184320,"data_stored":621125,"data_compressed":29962,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":10,"seq":42949672966,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27024,"kb_used_data":180,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344816,"statfs":{"total":96636764160,"available":96609091584,"internally_reserved":0,"allocated":184320,"data_stored":621125,"data_compressed":29962,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705671,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27020,"kb_used_data":176,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344820,"statfs":{"total":96636764160,"available":96609095680,"internally_reserved":0,"allocated":180224,"data_stored":621106,"data_compressed":29962,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29962,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-22T03:08:11.516 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-22T03:08:11.618 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:08:11.613+0000 7f94e122c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:08:11.621 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:08:11.620+0000 7f94e122c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:08:11.795 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-22T03:08:11.796 INFO:teuthology.orchestra.run.smithi153.stderr:dumped all 2024-09-22T03:08:11.808 INFO:teuthology.orchestra.run.smithi153.stdout:{"pg_ready":true,"pg_map":{"version":11,"stamp":"2024-09-22T03:08:10.104771+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":139,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":27,"acting":27,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":108020,"kb_used_data":656,"kb_used_omap":0,"kb_used_meta":107264,"kb_avail":377379340,"statfs":{"total":386547056640,"available":386436444160,"internally_reserved":0,"allocated":671744,"data_stored":1894113,"data_compressed":89886,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":109838336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.965670"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":16,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-22T03:08:07.154077+0000","last_change":"2024-09-22T03:08:07.157007+0000","last_active":"2024-09-22T03:08:07.154077+0000","last_peered":"2024-09-22T03:08:07.154077+0000","last_clean":"2024-09-22T03:08:07.154077+0000","last_became_active":"2024-09-22T03:08:05.192724+0000","last_became_peered":"2024-09-22T03:08:05.192724+0000","last_unstale":"2024-09-22T03:08:07.154077+0000","last_undegraded":"2024-09-22T03:08:07.154077+0000","last_fullsized":"2024-09-22T03:08:07.154077+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":16,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-22T03:08:07.154489+0000","last_change":"2024-09-22T03:08:07.156852+0000","last_active":"2024-09-22T03:08:07.154489+0000","last_peered":"2024-09-22T03:08:07.154489+0000","last_clean":"2024-09-22T03:08:07.154489+0000","last_became_active":"2024-09-22T03:08:05.203704+0000","last_became_peered":"2024-09-22T03:08:05.203704+0000","last_unstale":"2024-09-22T03:08:07.154489+0000","last_undegraded":"2024-09-22T03:08:07.154489+0000","last_fullsized":"2024-09-22T03:08:07.154489+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":16,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-22T03:08:07.155483+0000","last_change":"2024-09-22T03:08:07.156760+0000","last_active":"2024-09-22T03:08:07.155483+0000","last_peered":"2024-09-22T03:08:07.155483+0000","last_clean":"2024-09-22T03:08:07.155483+0000","last_became_active":"2024-09-22T03:08:05.189504+0000","last_became_peered":"2024-09-22T03:08:05.189504+0000","last_unstale":"2024-09-22T03:08:07.155483+0000","last_undegraded":"2024-09-22T03:08:07.155483+0000","last_fullsized":"2024-09-22T03:08:07.155483+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":16,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-22T03:08:07.152094+0000","last_change":"2024-09-22T03:08:07.153737+0000","last_active":"2024-09-22T03:08:07.152094+0000","last_peered":"2024-09-22T03:08:07.152094+0000","last_clean":"2024-09-22T03:08:07.152094+0000","last_became_active":"2024-09-22T03:08:05.184155+0000","last_became_peered":"2024-09-22T03:08:05.184155+0000","last_unstale":"2024-09-22T03:08:07.152094+0000","last_undegraded":"2024-09-22T03:08:07.152094+0000","last_fullsized":"2024-09-22T03:08:07.152094+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"12'97","reported_seq":142,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-22T03:08:07.155728+0000","last_change":"2024-09-22T03:08:05.205288+0000","last_active":"2024-09-22T03:08:07.155728+0000","last_peered":"2024-09-22T03:08:07.155728+0000","last_clean":"2024-09-22T03:08:07.155728+0000","last_became_active":"2024-09-22T03:08:05.195537+0000","last_became_peered":"2024-09-22T03:08:05.195537+0000","last_unstale":"2024-09-22T03:08:07.155728+0000","last_undegraded":"2024-09-22T03:08:07.155728+0000","last_fullsized":"2024-09-22T03:08:07.155728+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-22T03:08:07.155274+0000","last_change":"2024-09-22T03:08:07.156930+0000","last_active":"2024-09-22T03:08:07.155274+0000","last_peered":"2024-09-22T03:08:07.155274+0000","last_clean":"2024-09-22T03:08:07.155274+0000","last_became_active":"2024-09-22T03:08:05.183326+0000","last_became_peered":"2024-09-22T03:08:05.183326+0000","last_unstale":"2024-09-22T03:08:07.155274+0000","last_undegraded":"2024-09-22T03:08:07.155274+0000","last_fullsized":"2024-09-22T03:08:07.155274+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"13'1","reported_seq":18,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-22T03:08:07.155023+0000","last_change":"2024-09-22T03:08:07.155023+0000","last_active":"2024-09-22T03:08:07.155023+0000","last_peered":"2024-09-22T03:08:07.155023+0000","last_clean":"2024-09-22T03:08:07.155023+0000","last_became_active":"2024-09-22T03:08:05.190985+0000","last_became_peered":"2024-09-22T03:08:05.190985+0000","last_unstale":"2024-09-22T03:08:07.155023+0000","last_undegraded":"2024-09-22T03:08:07.155023+0000","last_fullsized":"2024-09-22T03:08:07.155023+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":16,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-22T03:08:07.153136+0000","last_change":"2024-09-22T03:08:07.155571+0000","last_active":"2024-09-22T03:08:07.153136+0000","last_peered":"2024-09-22T03:08:07.153136+0000","last_clean":"2024-09-22T03:08:07.153136+0000","last_became_active":"2024-09-22T03:08:05.189949+0000","last_became_peered":"2024-09-22T03:08:05.189949+0000","last_unstale":"2024-09-22T03:08:07.153136+0000","last_undegraded":"2024-09-22T03:08:07.153136+0000","last_fullsized":"2024-09-22T03:08:07.153136+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":16,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-22T03:08:07.152054+0000","last_change":"2024-09-22T03:08:07.154145+0000","last_active":"2024-09-22T03:08:07.152054+0000","last_peered":"2024-09-22T03:08:07.152054+0000","last_clean":"2024-09-22T03:08:07.152054+0000","last_became_active":"2024-09-22T03:08:05.198559+0000","last_became_peered":"2024-09-22T03:08:05.198559+0000","last_unstale":"2024-09-22T03:08:07.152054+0000","last_undegraded":"2024-09-22T03:08:07.152054+0000","last_fullsized":"2024-09-22T03:08:07.152054+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29962,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":1},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":24,"acting":24,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":10,"seq":42949672966,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26952,"kb_used_data":120,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344888,"statfs":{"total":96636764160,"available":96609165312,"internally_reserved":0,"allocated":122880,"data_stored":30757,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":10,"seq":42949672966,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27024,"kb_used_data":180,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344816,"statfs":{"total":96636764160,"available":96609091584,"internally_reserved":0,"allocated":184320,"data_stored":621125,"data_compressed":29962,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":10,"seq":42949672966,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27024,"kb_used_data":180,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344816,"statfs":{"total":96636764160,"available":96609091584,"internally_reserved":0,"allocated":184320,"data_stored":621125,"data_compressed":29962,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705671,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27020,"kb_used_data":176,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344820,"statfs":{"total":96636764160,"available":96609095680,"internally_reserved":0,"allocated":180224,"data_stored":621106,"data_compressed":29962,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29962,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-22T03:08:11.809 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-22T03:08:11.809 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-09-22T03:08:11.809 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-09-22T03:08:11.809 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-09-22T03:08:11.911 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:08:11.907+0000 7f5071197640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:08:11.914 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:08:11.912+0000 7f5071197640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:08:12.127 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-22T03:08:12.127 INFO:teuthology.orchestra.run.smithi153.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-22T03:08:12.140 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-09-22T03:08:12.141 INFO:teuthology.run_tasks:Running task thrashosds... 2024-09-22T03:08:12.151 INFO:tasks.thrashosds:config is {'ceph_objectstore_tool': False, 'chance_inject_pause_short': 0, 'chance_reset_purged_snaps_last': 0, 'chance_thrash_cluster_full': 0, 'chance_thrash_pg_upmap': 0, 'dump_ops_enable': False, 'min_in': 3, 'noscrub_toggle_delay': 0, 'reweight_osd': 0, 'sighup_delay': 0, 'thrash_primary_affinity': False, 'timeout': 2400, 'optrack_toggle_delay': 2.0, 'random_eio': 0.0} 2024-09-22T03:08:12.151 INFO:tasks.thrashosds:overrides is {'admin_socket': {'branch': 'squid'}, 'ceph': {'conf': {'client.0': {'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc', 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 3}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'osd pool default crimson': True, 'paxos service trim min': 10}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd failsafe full ratio': 0.95, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}, 'osd.0': {'crimson seastar cpu cores': '0-2'}, 'osd.1': {'crimson seastar cpu cores': '3-5'}, 'osd.2': {'crimson seastar cpu cores': '0-2'}, 'osd.3': {'crimson seastar cpu cores': '3-5'}}, 'flavor': 'crimson', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum', 'noscrub', 'nodeep-scrub'], 'sha1': '4e45e6b490e6cbe082cdf155fea780438ec53fc7'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'crimson', 'sha1': '4e45e6b490e6cbe082cdf155fea780438ec53fc7'}}, 'selinux': {'allowlist': ['scontext=system_u:system_r:getty_t:s0']}, 'thrashosds': {'min_in': 4}, 'workunit': {'branch': 'squid', 'env': {'CRIMSON_COMPAT': '1'}, 'sha1': '4e45e6b490e6cbe082cdf155fea780438ec53fc7'}} 2024-09-22T03:08:12.151 INFO:tasks.thrashosds:config is {'ceph_objectstore_tool': False, 'chance_inject_pause_short': 0, 'chance_reset_purged_snaps_last': 0, 'chance_thrash_cluster_full': 0, 'chance_thrash_pg_upmap': 0, 'dump_ops_enable': False, 'min_in': 4, 'noscrub_toggle_delay': 0, 'reweight_osd': 0, 'sighup_delay': 0, 'thrash_primary_affinity': False, 'timeout': 2400, 'optrack_toggle_delay': 2.0, 'random_eio': 0.0} 2024-09-22T03:08:12.151 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr_debug_aggressive_pg_num_changes true 2024-09-22T03:08:12.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:08:12.251+0000 7fee55b57640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:08:12.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:08:12.261+0000 7fee55b57640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:08:12.454 INFO:tasks.thrashosds:Beginning thrashosds... 2024-09-22T03:08:12.454 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-22T03:08:12.454 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-22T03:08:12.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:08:12.553+0000 7f1d5de58640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:08:12.561 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:08:12.560+0000 7f1d5de58640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:08:12.734 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-22T03:08:12.735 INFO:teuthology.orchestra.run.smithi153.stderr:dumped all 2024-09-22T03:08:12.747 INFO:teuthology.orchestra.run.smithi153.stdout:{"pg_ready":true,"pg_map":{"version":12,"stamp":"2024-09-22T03:08:12.105221+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":139,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":27,"acting":27,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":108020,"kb_used_data":656,"kb_used_omap":0,"kb_used_meta":107264,"kb_avail":377379340,"statfs":{"total":386547056640,"available":386436444160,"internally_reserved":0,"allocated":671744,"data_stored":1894113,"data_compressed":89886,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":109838336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.966120"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":16,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-22T03:08:07.154077+0000","last_change":"2024-09-22T03:08:07.157007+0000","last_active":"2024-09-22T03:08:07.154077+0000","last_peered":"2024-09-22T03:08:07.154077+0000","last_clean":"2024-09-22T03:08:07.154077+0000","last_became_active":"2024-09-22T03:08:05.192724+0000","last_became_peered":"2024-09-22T03:08:05.192724+0000","last_unstale":"2024-09-22T03:08:07.154077+0000","last_undegraded":"2024-09-22T03:08:07.154077+0000","last_fullsized":"2024-09-22T03:08:07.154077+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":16,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-22T03:08:07.154489+0000","last_change":"2024-09-22T03:08:07.156852+0000","last_active":"2024-09-22T03:08:07.154489+0000","last_peered":"2024-09-22T03:08:07.154489+0000","last_clean":"2024-09-22T03:08:07.154489+0000","last_became_active":"2024-09-22T03:08:05.203704+0000","last_became_peered":"2024-09-22T03:08:05.203704+0000","last_unstale":"2024-09-22T03:08:07.154489+0000","last_undegraded":"2024-09-22T03:08:07.154489+0000","last_fullsized":"2024-09-22T03:08:07.154489+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":16,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-22T03:08:07.155483+0000","last_change":"2024-09-22T03:08:07.156760+0000","last_active":"2024-09-22T03:08:07.155483+0000","last_peered":"2024-09-22T03:08:07.155483+0000","last_clean":"2024-09-22T03:08:07.155483+0000","last_became_active":"2024-09-22T03:08:05.189504+0000","last_became_peered":"2024-09-22T03:08:05.189504+0000","last_unstale":"2024-09-22T03:08:07.155483+0000","last_undegraded":"2024-09-22T03:08:07.155483+0000","last_fullsized":"2024-09-22T03:08:07.155483+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":16,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-22T03:08:07.152094+0000","last_change":"2024-09-22T03:08:07.153737+0000","last_active":"2024-09-22T03:08:07.152094+0000","last_peered":"2024-09-22T03:08:07.152094+0000","last_clean":"2024-09-22T03:08:07.152094+0000","last_became_active":"2024-09-22T03:08:05.184155+0000","last_became_peered":"2024-09-22T03:08:05.184155+0000","last_unstale":"2024-09-22T03:08:07.152094+0000","last_undegraded":"2024-09-22T03:08:07.152094+0000","last_fullsized":"2024-09-22T03:08:07.152094+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"12'97","reported_seq":142,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-22T03:08:07.155728+0000","last_change":"2024-09-22T03:08:05.205288+0000","last_active":"2024-09-22T03:08:07.155728+0000","last_peered":"2024-09-22T03:08:07.155728+0000","last_clean":"2024-09-22T03:08:07.155728+0000","last_became_active":"2024-09-22T03:08:05.195537+0000","last_became_peered":"2024-09-22T03:08:05.195537+0000","last_unstale":"2024-09-22T03:08:07.155728+0000","last_undegraded":"2024-09-22T03:08:07.155728+0000","last_fullsized":"2024-09-22T03:08:07.155728+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-22T03:08:07.155274+0000","last_change":"2024-09-22T03:08:07.156930+0000","last_active":"2024-09-22T03:08:07.155274+0000","last_peered":"2024-09-22T03:08:07.155274+0000","last_clean":"2024-09-22T03:08:07.155274+0000","last_became_active":"2024-09-22T03:08:05.183326+0000","last_became_peered":"2024-09-22T03:08:05.183326+0000","last_unstale":"2024-09-22T03:08:07.155274+0000","last_undegraded":"2024-09-22T03:08:07.155274+0000","last_fullsized":"2024-09-22T03:08:07.155274+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"13'1","reported_seq":18,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-22T03:08:07.155023+0000","last_change":"2024-09-22T03:08:07.155023+0000","last_active":"2024-09-22T03:08:07.155023+0000","last_peered":"2024-09-22T03:08:07.155023+0000","last_clean":"2024-09-22T03:08:07.155023+0000","last_became_active":"2024-09-22T03:08:05.190985+0000","last_became_peered":"2024-09-22T03:08:05.190985+0000","last_unstale":"2024-09-22T03:08:07.155023+0000","last_undegraded":"2024-09-22T03:08:07.155023+0000","last_fullsized":"2024-09-22T03:08:07.155023+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":16,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-22T03:08:07.153136+0000","last_change":"2024-09-22T03:08:07.155571+0000","last_active":"2024-09-22T03:08:07.153136+0000","last_peered":"2024-09-22T03:08:07.153136+0000","last_clean":"2024-09-22T03:08:07.153136+0000","last_became_active":"2024-09-22T03:08:05.189949+0000","last_became_peered":"2024-09-22T03:08:05.189949+0000","last_unstale":"2024-09-22T03:08:07.153136+0000","last_undegraded":"2024-09-22T03:08:07.153136+0000","last_fullsized":"2024-09-22T03:08:07.153136+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":16,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-22T03:08:07.152054+0000","last_change":"2024-09-22T03:08:07.154145+0000","last_active":"2024-09-22T03:08:07.152054+0000","last_peered":"2024-09-22T03:08:07.152054+0000","last_clean":"2024-09-22T03:08:07.152054+0000","last_became_active":"2024-09-22T03:08:05.198559+0000","last_became_peered":"2024-09-22T03:08:05.198559+0000","last_unstale":"2024-09-22T03:08:07.152054+0000","last_undegraded":"2024-09-22T03:08:07.152054+0000","last_fullsized":"2024-09-22T03:08:07.152054+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29962,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":1},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":24,"acting":24,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":10,"seq":42949672966,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26952,"kb_used_data":120,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344888,"statfs":{"total":96636764160,"available":96609165312,"internally_reserved":0,"allocated":122880,"data_stored":30757,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":10,"seq":42949672966,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27024,"kb_used_data":180,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344816,"statfs":{"total":96636764160,"available":96609091584,"internally_reserved":0,"allocated":184320,"data_stored":621125,"data_compressed":29962,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":10,"seq":42949672966,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27024,"kb_used_data":180,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344816,"statfs":{"total":96636764160,"available":96609091584,"internally_reserved":0,"allocated":184320,"data_stored":621125,"data_compressed":29962,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705673,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27020,"kb_used_data":176,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344820,"statfs":{"total":96636764160,"available":96609095680,"internally_reserved":0,"allocated":180224,"data_stored":621106,"data_compressed":29962,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29962,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-22T03:08:12.748 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-22T03:08:12.853 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:08:12.848+0000 7f7842397640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:08:12.855 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:08:12.854+0000 7f7842397640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:08:13.030 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-22T03:08:13.030 INFO:teuthology.orchestra.run.smithi153.stderr:dumped all 2024-09-22T03:08:13.043 INFO:teuthology.orchestra.run.smithi153.stdout:{"pg_ready":true,"pg_map":{"version":12,"stamp":"2024-09-22T03:08:12.105221+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":139,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":27,"acting":27,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":108020,"kb_used_data":656,"kb_used_omap":0,"kb_used_meta":107264,"kb_avail":377379340,"statfs":{"total":386547056640,"available":386436444160,"internally_reserved":0,"allocated":671744,"data_stored":1894113,"data_compressed":89886,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":109838336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.966120"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":16,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-22T03:08:07.154077+0000","last_change":"2024-09-22T03:08:07.157007+0000","last_active":"2024-09-22T03:08:07.154077+0000","last_peered":"2024-09-22T03:08:07.154077+0000","last_clean":"2024-09-22T03:08:07.154077+0000","last_became_active":"2024-09-22T03:08:05.192724+0000","last_became_peered":"2024-09-22T03:08:05.192724+0000","last_unstale":"2024-09-22T03:08:07.154077+0000","last_undegraded":"2024-09-22T03:08:07.154077+0000","last_fullsized":"2024-09-22T03:08:07.154077+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":16,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-22T03:08:07.154489+0000","last_change":"2024-09-22T03:08:07.156852+0000","last_active":"2024-09-22T03:08:07.154489+0000","last_peered":"2024-09-22T03:08:07.154489+0000","last_clean":"2024-09-22T03:08:07.154489+0000","last_became_active":"2024-09-22T03:08:05.203704+0000","last_became_peered":"2024-09-22T03:08:05.203704+0000","last_unstale":"2024-09-22T03:08:07.154489+0000","last_undegraded":"2024-09-22T03:08:07.154489+0000","last_fullsized":"2024-09-22T03:08:07.154489+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":16,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-22T03:08:07.155483+0000","last_change":"2024-09-22T03:08:07.156760+0000","last_active":"2024-09-22T03:08:07.155483+0000","last_peered":"2024-09-22T03:08:07.155483+0000","last_clean":"2024-09-22T03:08:07.155483+0000","last_became_active":"2024-09-22T03:08:05.189504+0000","last_became_peered":"2024-09-22T03:08:05.189504+0000","last_unstale":"2024-09-22T03:08:07.155483+0000","last_undegraded":"2024-09-22T03:08:07.155483+0000","last_fullsized":"2024-09-22T03:08:07.155483+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":16,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-22T03:08:07.152094+0000","last_change":"2024-09-22T03:08:07.153737+0000","last_active":"2024-09-22T03:08:07.152094+0000","last_peered":"2024-09-22T03:08:07.152094+0000","last_clean":"2024-09-22T03:08:07.152094+0000","last_became_active":"2024-09-22T03:08:05.184155+0000","last_became_peered":"2024-09-22T03:08:05.184155+0000","last_unstale":"2024-09-22T03:08:07.152094+0000","last_undegraded":"2024-09-22T03:08:07.152094+0000","last_fullsized":"2024-09-22T03:08:07.152094+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"12'97","reported_seq":142,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-22T03:08:07.155728+0000","last_change":"2024-09-22T03:08:05.205288+0000","last_active":"2024-09-22T03:08:07.155728+0000","last_peered":"2024-09-22T03:08:07.155728+0000","last_clean":"2024-09-22T03:08:07.155728+0000","last_became_active":"2024-09-22T03:08:05.195537+0000","last_became_peered":"2024-09-22T03:08:05.195537+0000","last_unstale":"2024-09-22T03:08:07.155728+0000","last_undegraded":"2024-09-22T03:08:07.155728+0000","last_fullsized":"2024-09-22T03:08:07.155728+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-22T03:08:07.155274+0000","last_change":"2024-09-22T03:08:07.156930+0000","last_active":"2024-09-22T03:08:07.155274+0000","last_peered":"2024-09-22T03:08:07.155274+0000","last_clean":"2024-09-22T03:08:07.155274+0000","last_became_active":"2024-09-22T03:08:05.183326+0000","last_became_peered":"2024-09-22T03:08:05.183326+0000","last_unstale":"2024-09-22T03:08:07.155274+0000","last_undegraded":"2024-09-22T03:08:07.155274+0000","last_fullsized":"2024-09-22T03:08:07.155274+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"13'1","reported_seq":18,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-22T03:08:07.155023+0000","last_change":"2024-09-22T03:08:07.155023+0000","last_active":"2024-09-22T03:08:07.155023+0000","last_peered":"2024-09-22T03:08:07.155023+0000","last_clean":"2024-09-22T03:08:07.155023+0000","last_became_active":"2024-09-22T03:08:05.190985+0000","last_became_peered":"2024-09-22T03:08:05.190985+0000","last_unstale":"2024-09-22T03:08:07.155023+0000","last_undegraded":"2024-09-22T03:08:07.155023+0000","last_fullsized":"2024-09-22T03:08:07.155023+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":16,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-22T03:08:07.153136+0000","last_change":"2024-09-22T03:08:07.155571+0000","last_active":"2024-09-22T03:08:07.153136+0000","last_peered":"2024-09-22T03:08:07.153136+0000","last_clean":"2024-09-22T03:08:07.153136+0000","last_became_active":"2024-09-22T03:08:05.189949+0000","last_became_peered":"2024-09-22T03:08:05.189949+0000","last_unstale":"2024-09-22T03:08:07.153136+0000","last_undegraded":"2024-09-22T03:08:07.153136+0000","last_fullsized":"2024-09-22T03:08:07.153136+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":16,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-22T03:08:07.152054+0000","last_change":"2024-09-22T03:08:07.154145+0000","last_active":"2024-09-22T03:08:07.152054+0000","last_peered":"2024-09-22T03:08:07.152054+0000","last_clean":"2024-09-22T03:08:07.152054+0000","last_became_active":"2024-09-22T03:08:05.198559+0000","last_became_peered":"2024-09-22T03:08:05.198559+0000","last_unstale":"2024-09-22T03:08:07.152054+0000","last_undegraded":"2024-09-22T03:08:07.152054+0000","last_fullsized":"2024-09-22T03:08:07.152054+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29962,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":1},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":24,"acting":24,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":10,"seq":42949672966,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26952,"kb_used_data":120,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344888,"statfs":{"total":96636764160,"available":96609165312,"internally_reserved":0,"allocated":122880,"data_stored":30757,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":10,"seq":42949672966,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27024,"kb_used_data":180,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344816,"statfs":{"total":96636764160,"available":96609091584,"internally_reserved":0,"allocated":184320,"data_stored":621125,"data_compressed":29962,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":10,"seq":42949672966,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27024,"kb_used_data":180,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344816,"statfs":{"total":96636764160,"available":96609091584,"internally_reserved":0,"allocated":184320,"data_stored":621125,"data_compressed":29962,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705673,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27020,"kb_used_data":176,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344820,"statfs":{"total":96636764160,"available":96609095680,"internally_reserved":0,"allocated":180224,"data_stored":621106,"data_compressed":29962,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29962,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-22T03:08:13.043 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-22T03:08:13.044 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump 2024-09-22T03:08:13.150 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:08:13.146+0000 7f8ea386c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:08:13.154 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:08:13.152+0000 7f8ea386c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:08:13.335 INFO:teuthology.orchestra.run.smithi153.stdout:epoch 14 2024-09-22T03:08:13.336 INFO:teuthology.orchestra.run.smithi153.stdout:fsid f17abef2-bc08-46d2-abf1-d47a95f0e2ee 2024-09-22T03:08:13.336 INFO:teuthology.orchestra.run.smithi153.stdout:created 2024-09-22T03:07:53.859391+0000 2024-09-22T03:08:13.336 INFO:teuthology.orchestra.run.smithi153.stdout:modified 2024-09-22T03:08:07.141344+0000 2024-09-22T03:08:13.336 INFO:teuthology.orchestra.run.smithi153.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-09-22T03:08:13.336 INFO:teuthology.orchestra.run.smithi153.stdout:crush_version 4 2024-09-22T03:08:13.336 INFO:teuthology.orchestra.run.smithi153.stdout:full_ratio 0.95 2024-09-22T03:08:13.336 INFO:teuthology.orchestra.run.smithi153.stdout:backfillfull_ratio 0.9 2024-09-22T03:08:13.336 INFO:teuthology.orchestra.run.smithi153.stdout:nearfull_ratio 0.85 2024-09-22T03:08:13.336 INFO:teuthology.orchestra.run.smithi153.stdout:require_min_compat_client luminous 2024-09-22T03:08:13.336 INFO:teuthology.orchestra.run.smithi153.stdout:min_compat_client jewel 2024-09-22T03:08:13.336 INFO:teuthology.orchestra.run.smithi153.stdout:require_osd_release squid 2024-09-22T03:08:13.337 INFO:teuthology.orchestra.run.smithi153.stdout:stretch_mode_enabled false 2024-09-22T03:08:13.337 INFO:teuthology.orchestra.run.smithi153.stdout:allow_crimson=true 2024-09-22T03:08:13.337 INFO:teuthology.orchestra.run.smithi153.stdout:pool 1 'rbd' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 autoscale_mode off last_change 14 flags hashpspool,nopgchange,selfmanaged_snaps,crimson stripe_width 0 application rbd read_balance_score 1.50 2024-09-22T03:08:13.337 INFO:teuthology.orchestra.run.smithi153.stdout: removed_snaps_queue [2~1] 2024-09-22T03:08:13.337 INFO:teuthology.orchestra.run.smithi153.stdout:pool 2 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 13 flags hashpspool,nopgchange,crimson stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 4.00 2024-09-22T03:08:13.337 INFO:teuthology.orchestra.run.smithi153.stdout:max_osd 4 2024-09-22T03:08:13.337 INFO:teuthology.orchestra.run.smithi153.stdout:osd.0 up in weight 1 up_from 9 up_thru 11 down_at 0 last_clean_interval [0,0) v2:172.21.15.153:6805/406291079 v2:172.21.15.153:6804/406291079 exists,up 35bc7c89-cb7c-4664-bed6-4088394939d9 2024-09-22T03:08:13.337 INFO:teuthology.orchestra.run.smithi153.stdout:osd.1 up in weight 1 up_from 10 up_thru 11 down_at 0 last_clean_interval [0,0) v2:172.21.15.153:6803/1323003415 v2:172.21.15.153:6806/1323003415 exists,up 513ab51d-4bac-4f1a-ade1-755e26e4c8e9 2024-09-22T03:08:13.337 INFO:teuthology.orchestra.run.smithi153.stdout:osd.2 up in weight 1 up_from 10 up_thru 11 down_at 0 last_clean_interval [0,0) v2:172.21.15.196:6806/527153512 v2:172.21.15.196:6805/527153512 exists,up dd57bc3a-b76e-4c7f-8935-6c2b7eb8ef7b 2024-09-22T03:08:13.337 INFO:teuthology.orchestra.run.smithi153.stdout:osd.3 up in weight 1 up_from 10 up_thru 11 down_at 0 last_clean_interval [0,0) v2:172.21.15.196:6804/3375769528 v2:172.21.15.196:6803/3375769528 exists,up 76e7924f-5812-47ee-a5ee-ec837d12f327 2024-09-22T03:08:13.348 INFO:tasks.ceph.ceph_manager.ceph:['osd.0 up in weight 1 up_from 9 up_thru 11 down_at 0 last_clean_interval [0,0) v2:172.21.15.153:6805/406291079 v2:172.21.15.153:6804/406291079 exists,up 35bc7c89-cb7c-4664-bed6-4088394939d9', 'osd.1 up in weight 1 up_from 10 up_thru 11 down_at 0 last_clean_interval [0,0) v2:172.21.15.153:6803/1323003415 v2:172.21.15.153:6806/1323003415 exists,up 513ab51d-4bac-4f1a-ade1-755e26e4c8e9', 'osd.2 up in weight 1 up_from 10 up_thru 11 down_at 0 last_clean_interval [0,0) v2:172.21.15.196:6806/527153512 v2:172.21.15.196:6805/527153512 exists,up dd57bc3a-b76e-4c7f-8935-6c2b7eb8ef7b', 'osd.3 up in weight 1 up_from 10 up_thru 11 down_at 0 last_clean_interval [0,0) v2:172.21.15.196:6804/3375769528 v2:172.21.15.196:6803/3375769528 exists,up 76e7924f-5812-47ee-a5ee-ec837d12f327'] 2024-09-22T03:08:13.349 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mon.a.asok config show 2024-09-22T03:08:13.452 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-22T03:08:13.453 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "mon.a", 2024-09-22T03:08:13.453 INFO:teuthology.orchestra.run.smithi153.stdout: "cluster": "ceph", 2024-09-22T03:08:13.453 INFO:teuthology.orchestra.run.smithi153.stdout: "admin_socket": "/var/run/ceph/ceph-mon.a.asok", 2024-09-22T03:08:13.453 INFO:teuthology.orchestra.run.smithi153.stdout: "admin_socket_mode": "", 2024-09-22T03:08:13.453 INFO:teuthology.orchestra.run.smithi153.stdout: "auth_allow_insecure_global_id_reclaim": "true", 2024-09-22T03:08:13.453 INFO:teuthology.orchestra.run.smithi153.stdout: "auth_client_required": "cephx, none", 2024-09-22T03:08:13.453 INFO:teuthology.orchestra.run.smithi153.stdout: "auth_cluster_required": "cephx", 2024-09-22T03:08:13.453 INFO:teuthology.orchestra.run.smithi153.stdout: "auth_debug": "true", 2024-09-22T03:08:13.453 INFO:teuthology.orchestra.run.smithi153.stdout: "auth_expose_insecure_global_id_reclaim": "true", 2024-09-22T03:08:13.453 INFO:teuthology.orchestra.run.smithi153.stdout: "auth_mon_ticket_ttl": "660.000000", 2024-09-22T03:08:13.453 INFO:teuthology.orchestra.run.smithi153.stdout: "auth_service_required": "cephx", 2024-09-22T03:08:13.453 INFO:teuthology.orchestra.run.smithi153.stdout: "auth_service_ticket_ttl": "240.000000", 2024-09-22T03:08:13.454 INFO:teuthology.orchestra.run.smithi153.stdout: "auth_supported": "cephx", 2024-09-22T03:08:13.454 INFO:teuthology.orchestra.run.smithi153.stdout: "bdev_aio": "true", 2024-09-22T03:08:13.454 INFO:teuthology.orchestra.run.smithi153.stdout: "bdev_aio_max_queue_depth": "1024", 2024-09-22T03:08:13.454 INFO:teuthology.orchestra.run.smithi153.stdout: "bdev_aio_poll_ms": "250", 2024-09-22T03:08:13.454 INFO:teuthology.orchestra.run.smithi153.stdout: "bdev_aio_reap_max": "16", 2024-09-22T03:08:13.454 INFO:teuthology.orchestra.run.smithi153.stdout: "bdev_async_discard": "false", 2024-09-22T03:08:13.454 INFO:teuthology.orchestra.run.smithi153.stdout: "bdev_block_size": "4096", 2024-09-22T03:08:13.454 INFO:teuthology.orchestra.run.smithi153.stdout: "bdev_debug_aio": "false", 2024-09-22T03:08:13.454 INFO:teuthology.orchestra.run.smithi153.stdout: "bdev_debug_aio_log_age": "5.000000", 2024-09-22T03:08:13.454 INFO:teuthology.orchestra.run.smithi153.stdout: "bdev_debug_aio_suicide_timeout": "60.000000", 2024-09-22T03:08:13.454 INFO:teuthology.orchestra.run.smithi153.stdout: "bdev_debug_inflight_ios": "false", 2024-09-22T03:08:13.454 INFO:teuthology.orchestra.run.smithi153.stdout: "bdev_enable_discard": "false", 2024-09-22T03:08:13.454 INFO:teuthology.orchestra.run.smithi153.stdout: "bdev_flock_retry": "3", 2024-09-22T03:08:13.455 INFO:teuthology.orchestra.run.smithi153.stdout: "bdev_flock_retry_interval": "0.100000", 2024-09-22T03:08:13.455 INFO:teuthology.orchestra.run.smithi153.stdout: "bdev_inject_crash": "0", 2024-09-22T03:08:13.455 INFO:teuthology.orchestra.run.smithi153.stdout: "bdev_inject_crash_flush_delay": "2", 2024-09-22T03:08:13.455 INFO:teuthology.orchestra.run.smithi153.stdout: "bdev_ioring": "false", 2024-09-22T03:08:13.455 INFO:teuthology.orchestra.run.smithi153.stdout: "bdev_ioring_hipri": "false", 2024-09-22T03:08:13.455 INFO:teuthology.orchestra.run.smithi153.stdout: "bdev_ioring_sqthread_poll": "false", 2024-09-22T03:08:13.455 INFO:teuthology.orchestra.run.smithi153.stdout: "bdev_nvme_unbind_from_kernel": "false", 2024-09-22T03:08:13.455 INFO:teuthology.orchestra.run.smithi153.stdout: "bdev_read_buffer_alignment": "4096", 2024-09-22T03:08:13.455 INFO:teuthology.orchestra.run.smithi153.stdout: "bdev_read_preallocated_huge_buffers": "", 2024-09-22T03:08:13.455 INFO:teuthology.orchestra.run.smithi153.stdout: "bdev_stalled_read_warn_lifetime": "86400", 2024-09-22T03:08:13.455 INFO:teuthology.orchestra.run.smithi153.stdout: "bdev_stalled_read_warn_threshold": "1", 2024-09-22T03:08:13.455 INFO:teuthology.orchestra.run.smithi153.stdout: "bdev_type": "", 2024-09-22T03:08:13.455 INFO:teuthology.orchestra.run.smithi153.stdout: "bluefs_alloc_size": "1048576", 2024-09-22T03:08:13.455 INFO:teuthology.orchestra.run.smithi153.stdout: "bluefs_allocator": "hybrid", 2024-09-22T03:08:13.456 INFO:teuthology.orchestra.run.smithi153.stdout: "bluefs_buffered_io": "true", 2024-09-22T03:08:13.456 INFO:teuthology.orchestra.run.smithi153.stdout: "bluefs_check_for_zeros": "false", 2024-09-22T03:08:13.456 INFO:teuthology.orchestra.run.smithi153.stdout: "bluefs_check_volume_selector_often": "false", 2024-09-22T03:08:13.456 INFO:teuthology.orchestra.run.smithi153.stdout: "bluefs_check_volume_selector_on_umount": "false", 2024-09-22T03:08:13.456 INFO:teuthology.orchestra.run.smithi153.stdout: "bluefs_compact_log_sync": "false", 2024-09-22T03:08:13.456 INFO:teuthology.orchestra.run.smithi153.stdout: "bluefs_failed_shared_alloc_cooldown": "600.000000", 2024-09-22T03:08:13.456 INFO:teuthology.orchestra.run.smithi153.stdout: "bluefs_log_compact_min_ratio": "5.000000", 2024-09-22T03:08:13.456 INFO:teuthology.orchestra.run.smithi153.stdout: "bluefs_log_compact_min_size": "16777216", 2024-09-22T03:08:13.456 INFO:teuthology.orchestra.run.smithi153.stdout: "bluefs_log_replay_check_allocations": "true", 2024-09-22T03:08:13.456 INFO:teuthology.orchestra.run.smithi153.stdout: "bluefs_max_log_runway": "4194304", 2024-09-22T03:08:13.456 INFO:teuthology.orchestra.run.smithi153.stdout: "bluefs_max_prefetch": "1048576", 2024-09-22T03:08:13.456 INFO:teuthology.orchestra.run.smithi153.stdout: "bluefs_min_flush_size": "524288", 2024-09-22T03:08:13.456 INFO:teuthology.orchestra.run.smithi153.stdout: "bluefs_min_log_runway": "1048576", 2024-09-22T03:08:13.456 INFO:teuthology.orchestra.run.smithi153.stdout: "bluefs_replay_recovery": "false", 2024-09-22T03:08:13.456 INFO:teuthology.orchestra.run.smithi153.stdout: "bluefs_replay_recovery_disable_compact": "false", 2024-09-22T03:08:13.457 INFO:teuthology.orchestra.run.smithi153.stdout: "bluefs_shared_alloc_size": "65536", 2024-09-22T03:08:13.457 INFO:teuthology.orchestra.run.smithi153.stdout: "bluefs_sync_write": "false", 2024-09-22T03:08:13.457 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_2q_cache_kin_ratio": "0.500000", 2024-09-22T03:08:13.457 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_2q_cache_kout_ratio": "0.500000", 2024-09-22T03:08:13.457 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_alloc_stats_dump_interval": "86400.000000", 2024-09-22T03:08:13.457 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_allocation_from_file": "true", 2024-09-22T03:08:13.457 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_allocator": "hybrid", 2024-09-22T03:08:13.457 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_avl_alloc_bf_free_pct": "4", 2024-09-22T03:08:13.457 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_avl_alloc_bf_threshold": "131072", 2024-09-22T03:08:13.457 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_avl_alloc_ff_max_search_bytes": "16777216", 2024-09-22T03:08:13.457 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_avl_alloc_ff_max_search_count": "100", 2024-09-22T03:08:13.457 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_bdev_label_multi": "true", 2024-09-22T03:08:13.457 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_bdev_label_multi_upgrade": "false", 2024-09-22T03:08:13.457 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_bdev_label_require_all": "true", 2024-09-22T03:08:13.457 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_bitmapallocator_blocks_per_zone": "1024", 2024-09-22T03:08:13.458 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_bitmapallocator_span_size": "1024", 2024-09-22T03:08:13.458 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_blobid_prealloc": "10240", 2024-09-22T03:08:13.458 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_block_create": "true", 2024-09-22T03:08:13.458 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_block_db_create": "false", 2024-09-22T03:08:13.458 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_block_db_path": "", 2024-09-22T03:08:13.458 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_block_db_size": "0", 2024-09-22T03:08:13.458 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_block_path": "", 2024-09-22T03:08:13.458 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_block_preallocate_file": "false", 2024-09-22T03:08:13.458 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_block_size": "107374182400", 2024-09-22T03:08:13.458 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_block_wal_create": "false", 2024-09-22T03:08:13.458 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_block_wal_path": "", 2024-09-22T03:08:13.458 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_block_wal_size": "100663296", 2024-09-22T03:08:13.458 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_bluefs": "true", 2024-09-22T03:08:13.458 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_bluefs_alloc_failure_dump_interval": "0.000000", 2024-09-22T03:08:13.459 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_bluefs_env_mirror": "false", 2024-09-22T03:08:13.459 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_bluefs_max_free": "10737418240", 2024-09-22T03:08:13.459 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_age_bin_interval": "1.000000", 2024-09-22T03:08:13.459 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_age_bins_data": "1 2 6 24 120 720 0 0 0 0", 2024-09-22T03:08:13.459 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_age_bins_kv": "1 2 6 24 120 720 0 0 0 0", 2024-09-22T03:08:13.459 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_age_bins_kv_onode": "0 0 0 0 0 0 0 0 0 720", 2024-09-22T03:08:13.459 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_age_bins_meta": "1 2 6 24 120 720 0 0 0 0", 2024-09-22T03:08:13.459 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_autotune": "true", 2024-09-22T03:08:13.459 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_autotune_interval": "5.000000", 2024-09-22T03:08:13.459 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_kv_onode_ratio": "0.040000", 2024-09-22T03:08:13.459 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_kv_ratio": "0.450000", 2024-09-22T03:08:13.459 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_meta_ratio": "0.450000", 2024-09-22T03:08:13.459 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_size": "0", 2024-09-22T03:08:13.460 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_size_hdd": "1073741824", 2024-09-22T03:08:13.460 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_size_ssd": "3221225472", 2024-09-22T03:08:13.460 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_trim_interval": "0.050000", 2024-09-22T03:08:13.460 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_trim_max_skip_pinned": "1000", 2024-09-22T03:08:13.460 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_type": "2q", 2024-09-22T03:08:13.460 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cleaner_sleep_interval": "5.000000", 2024-09-22T03:08:13.460 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_clone_cow": "true", 2024-09-22T03:08:13.460 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_compression_algorithm": "snappy", 2024-09-22T03:08:13.460 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_compression_max_blob_size": "0", 2024-09-22T03:08:13.460 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_compression_max_blob_size_hdd": "65536", 2024-09-22T03:08:13.460 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_compression_max_blob_size_ssd": "65536", 2024-09-22T03:08:13.460 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_compression_min_blob_size": "0", 2024-09-22T03:08:13.460 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_compression_min_blob_size_hdd": "8192", 2024-09-22T03:08:13.460 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_compression_min_blob_size_ssd": "65536", 2024-09-22T03:08:13.461 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_compression_mode": "none", 2024-09-22T03:08:13.461 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_compression_required_ratio": "0.875000", 2024-09-22T03:08:13.461 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_csum_type": "crc32c", 2024-09-22T03:08:13.461 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_debug_enforce_settings": "default", 2024-09-22T03:08:13.461 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_debug_freelist": "false", 2024-09-22T03:08:13.461 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_debug_fsck_abort": "false", 2024-09-22T03:08:13.461 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_debug_inject_allocation_from_file_failure": "0.000000", 2024-09-22T03:08:13.461 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_debug_inject_csum_err_probability": "0.000000", 2024-09-22T03:08:13.461 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_debug_inject_read_err": "false", 2024-09-22T03:08:13.461 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_debug_legacy_omap": "false", 2024-09-22T03:08:13.461 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_debug_no_reuse_blocks": "false", 2024-09-22T03:08:13.461 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_debug_omit_block_device_write": "false", 2024-09-22T03:08:13.461 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_debug_omit_kv_commit": "false", 2024-09-22T03:08:13.461 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_debug_permit_any_bdev_label": "false", 2024-09-22T03:08:13.462 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_debug_prefragment_max": "1048576", 2024-09-22T03:08:13.462 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_debug_random_read_err": "0.000000", 2024-09-22T03:08:13.462 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_debug_randomize_serial_transaction": "0", 2024-09-22T03:08:13.462 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_debug_small_allocations": "0", 2024-09-22T03:08:13.462 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_debug_too_many_blobs_threshold": "24576", 2024-09-22T03:08:13.462 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_default_buffered_read": "true", 2024-09-22T03:08:13.462 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_default_buffered_write": "false", 2024-09-22T03:08:13.462 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_deferred_batch_ops": "0", 2024-09-22T03:08:13.462 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_deferred_batch_ops_hdd": "64", 2024-09-22T03:08:13.462 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_deferred_batch_ops_ssd": "16", 2024-09-22T03:08:13.462 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_elastic_shared_blobs": "true", 2024-09-22T03:08:13.462 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_extent_map_inline_shard_prealloc_size": "256", 2024-09-22T03:08:13.462 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_extent_map_shard_max_size": "1200", 2024-09-22T03:08:13.463 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_extent_map_shard_min_size": "150", 2024-09-22T03:08:13.463 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_extent_map_shard_target_size": "500", 2024-09-22T03:08:13.463 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_extent_map_shard_target_size_slop": "0.200000", 2024-09-22T03:08:13.463 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_fail_eio": "false", 2024-09-22T03:08:13.463 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_freelist_blocks_per_key": "128", 2024-09-22T03:08:13.463 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_fsck_error_on_no_per_pg_omap": "false", 2024-09-22T03:08:13.463 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_fsck_error_on_no_per_pool_omap": "false", 2024-09-22T03:08:13.463 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_fsck_error_on_no_per_pool_stats": "false", 2024-09-22T03:08:13.463 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_fsck_on_mkfs": "true", 2024-09-22T03:08:13.463 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_fsck_on_mkfs_deep": "false", 2024-09-22T03:08:13.463 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_fsck_on_mount": "false", 2024-09-22T03:08:13.463 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_fsck_on_mount_deep": "false", 2024-09-22T03:08:13.463 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_fsck_on_umount": "false", 2024-09-22T03:08:13.463 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_fsck_on_umount_deep": "false", 2024-09-22T03:08:13.463 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_fsck_quick_fix_on_mount": "false", 2024-09-22T03:08:13.464 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_fsck_quick_fix_threads": "2", 2024-09-22T03:08:13.464 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_fsck_read_bytes_cap": "67108864", 2024-09-22T03:08:13.464 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_fsck_shared_blob_tracker_size": "0.031250", 2024-09-22T03:08:13.464 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_gc_enable_blob_threshold": "0", 2024-09-22T03:08:13.464 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_gc_enable_total_threshold": "0", 2024-09-22T03:08:13.464 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_hybrid_alloc_mem_cap": "67108864", 2024-09-22T03:08:13.464 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_ignore_data_csum": "false", 2024-09-22T03:08:13.464 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_kv_sync_util_logging_s": "10.000000", 2024-09-22T03:08:13.464 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_kvbackend": "rocksdb", 2024-09-22T03:08:13.464 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_log_collection_list_age": "60.000000", 2024-09-22T03:08:13.464 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_log_omap_iterator_age": "5.000000", 2024-09-22T03:08:13.464 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_log_op_age": "5.000000", 2024-09-22T03:08:13.465 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_max_alloc_size": "0", 2024-09-22T03:08:13.465 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_max_blob_size": "0", 2024-09-22T03:08:13.465 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_max_blob_size_hdd": "65536", 2024-09-22T03:08:13.465 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_max_blob_size_ssd": "65536", 2024-09-22T03:08:13.465 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_max_defer_interval": "3.000000", 2024-09-22T03:08:13.465 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_max_deferred_txc": "32", 2024-09-22T03:08:13.465 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_min_alloc_size": "0", 2024-09-22T03:08:13.465 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_min_alloc_size_hdd": "4096", 2024-09-22T03:08:13.465 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_min_alloc_size_ssd": "4096", 2024-09-22T03:08:13.465 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_nid_prealloc": "1024", 2024-09-22T03:08:13.465 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_prefer_deferred_size": "0", 2024-09-22T03:08:13.465 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_prefer_deferred_size_hdd": "65536", 2024-09-22T03:08:13.465 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_prefer_deferred_size_ssd": "0", 2024-09-22T03:08:13.465 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_qfsck_on_mount": "true", 2024-09-22T03:08:13.466 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_retry_disk_reads": "3", 2024-09-22T03:08:13.466 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_rocksdb_cf": "true", 2024-09-22T03:08:13.466 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_rocksdb_cfs": "m(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L=min_write_buffer_number_to_merge=32 P=min_write_buffer_number_to_merge=32", 2024-09-22T03:08:13.466 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_rocksdb_options": "compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0", 2024-09-22T03:08:13.466 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_rocksdb_options_annex": "", 2024-09-22T03:08:13.466 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_slow_ops_warn_lifetime": "86400", 2024-09-22T03:08:13.466 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_slow_ops_warn_threshold": "1", 2024-09-22T03:08:13.466 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_spdk_coremask": "0x1", 2024-09-22T03:08:13.466 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_spdk_io_sleep": "5", 2024-09-22T03:08:13.466 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_spdk_max_io_completion": "0", 2024-09-22T03:08:13.466 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_spdk_mem": "512", 2024-09-22T03:08:13.466 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_sync_submit_transaction": "false", 2024-09-22T03:08:13.466 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_throttle_bytes": "67108864", 2024-09-22T03:08:13.467 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_throttle_cost_per_io": "0", 2024-09-22T03:08:13.467 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_throttle_cost_per_io_hdd": "670000", 2024-09-22T03:08:13.467 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_throttle_cost_per_io_ssd": "4000", 2024-09-22T03:08:13.467 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_throttle_deferred_bytes": "134217728", 2024-09-22T03:08:13.467 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_throttle_trace_rate": "0.000000", 2024-09-22T03:08:13.467 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_tracing": "false", 2024-09-22T03:08:13.467 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_use_optimal_io_size_for_min_alloc_size": "false", 2024-09-22T03:08:13.467 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_volume_selection_policy": "use_some_extra", 2024-09-22T03:08:13.467 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_volume_selection_reserved": "0", 2024-09-22T03:08:13.467 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_volume_selection_reserved_factor": "2.000000", 2024-09-22T03:08:13.467 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_warn_on_bluefs_spillover": "true", 2024-09-22T03:08:13.467 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_warn_on_legacy_statfs": "true", 2024-09-22T03:08:13.467 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_warn_on_no_per_pg_omap": "false", 2024-09-22T03:08:13.467 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_warn_on_no_per_pool_omap": "true", 2024-09-22T03:08:13.468 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_warn_on_spurious_read_errors": "true", 2024-09-22T03:08:13.468 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_zero_block_detection": "false", 2024-09-22T03:08:13.468 INFO:teuthology.orchestra.run.smithi153.stdout: "cephadm_path": "/usr/sbin/cephadm", 2024-09-22T03:08:13.468 INFO:teuthology.orchestra.run.smithi153.stdout: "cephfs_mirror_action_update_interval": "2", 2024-09-22T03:08:13.468 INFO:teuthology.orchestra.run.smithi153.stdout: "cephfs_mirror_directory_scan_interval": "10", 2024-09-22T03:08:13.468 INFO:teuthology.orchestra.run.smithi153.stdout: "cephfs_mirror_max_concurrent_directory_syncs": "3", 2024-09-22T03:08:13.468 INFO:teuthology.orchestra.run.smithi153.stdout: "cephfs_mirror_max_consecutive_failures_per_directory": "10", 2024-09-22T03:08:13.468 INFO:teuthology.orchestra.run.smithi153.stdout: "cephfs_mirror_max_snapshot_sync_per_cycle": "3", 2024-09-22T03:08:13.468 INFO:teuthology.orchestra.run.smithi153.stdout: "cephfs_mirror_mount_timeout": "10", 2024-09-22T03:08:13.468 INFO:teuthology.orchestra.run.smithi153.stdout: "cephfs_mirror_perf_stats_prio": "5", 2024-09-22T03:08:13.468 INFO:teuthology.orchestra.run.smithi153.stdout: "cephfs_mirror_restart_mirror_on_blocklist_interval": "30", 2024-09-22T03:08:13.468 INFO:teuthology.orchestra.run.smithi153.stdout: "cephfs_mirror_restart_mirror_on_failure_interval": "20", 2024-09-22T03:08:13.468 INFO:teuthology.orchestra.run.smithi153.stdout: "cephfs_mirror_retry_failed_directories_interval": "60", 2024-09-22T03:08:13.468 INFO:teuthology.orchestra.run.smithi153.stdout: "cephsqlite_blocklist_dead_locker": "true", 2024-09-22T03:08:13.468 INFO:teuthology.orchestra.run.smithi153.stdout: "cephsqlite_lock_renewal_interval": "2000", 2024-09-22T03:08:13.469 INFO:teuthology.orchestra.run.smithi153.stdout: "cephsqlite_lock_renewal_timeout": "30000", 2024-09-22T03:08:13.469 INFO:teuthology.orchestra.run.smithi153.stdout: "cephx_cluster_require_signatures": "false", 2024-09-22T03:08:13.469 INFO:teuthology.orchestra.run.smithi153.stdout: "cephx_cluster_require_version": "2", 2024-09-22T03:08:13.469 INFO:teuthology.orchestra.run.smithi153.stdout: "cephx_require_signatures": "false", 2024-09-22T03:08:13.469 INFO:teuthology.orchestra.run.smithi153.stdout: "cephx_require_version": "2", 2024-09-22T03:08:13.469 INFO:teuthology.orchestra.run.smithi153.stdout: "cephx_service_require_signatures": "false", 2024-09-22T03:08:13.469 INFO:teuthology.orchestra.run.smithi153.stdout: "cephx_service_require_version": "2", 2024-09-22T03:08:13.469 INFO:teuthology.orchestra.run.smithi153.stdout: "cephx_sign_messages": "true", 2024-09-22T03:08:13.469 INFO:teuthology.orchestra.run.smithi153.stdout: "chdir": "", 2024-09-22T03:08:13.469 INFO:teuthology.orchestra.run.smithi153.stdout: "client_acl_type": "", 2024-09-22T03:08:13.469 INFO:teuthology.orchestra.run.smithi153.stdout: "client_asio_thread_count": "2", 2024-09-22T03:08:13.469 INFO:teuthology.orchestra.run.smithi153.stdout: "client_cache_mid": "0.750000", 2024-09-22T03:08:13.469 INFO:teuthology.orchestra.run.smithi153.stdout: "client_cache_size": "16384", 2024-09-22T03:08:13.469 INFO:teuthology.orchestra.run.smithi153.stdout: "client_caps_release_delay": "5", 2024-09-22T03:08:13.470 INFO:teuthology.orchestra.run.smithi153.stdout: "client_check_pool_perm": "true", 2024-09-22T03:08:13.470 INFO:teuthology.orchestra.run.smithi153.stdout: "client_collect_and_send_global_metrics": "false", 2024-09-22T03:08:13.470 INFO:teuthology.orchestra.run.smithi153.stdout: "client_debug_force_sync_read": "false", 2024-09-22T03:08:13.470 INFO:teuthology.orchestra.run.smithi153.stdout: "client_debug_getattr_caps": "false", 2024-09-22T03:08:13.470 INFO:teuthology.orchestra.run.smithi153.stdout: "client_debug_inject_features": "", 2024-09-22T03:08:13.470 INFO:teuthology.orchestra.run.smithi153.stdout: "client_debug_inject_tick_delay": "0", 2024-09-22T03:08:13.470 INFO:teuthology.orchestra.run.smithi153.stdout: "client_die_on_failed_dentry_invalidate": "true", 2024-09-22T03:08:13.470 INFO:teuthology.orchestra.run.smithi153.stdout: "client_die_on_failed_remount": "false", 2024-09-22T03:08:13.470 INFO:teuthology.orchestra.run.smithi153.stdout: "client_dirsize_rbytes": "true", 2024-09-22T03:08:13.470 INFO:teuthology.orchestra.run.smithi153.stdout: "client_force_lazyio": "false", 2024-09-22T03:08:13.470 INFO:teuthology.orchestra.run.smithi153.stdout: "client_fs": "", 2024-09-22T03:08:13.470 INFO:teuthology.orchestra.run.smithi153.stdout: "client_inject_fixed_oldest_tid": "false", 2024-09-22T03:08:13.470 INFO:teuthology.orchestra.run.smithi153.stdout: "client_inject_release_failure": "false", 2024-09-22T03:08:13.470 INFO:teuthology.orchestra.run.smithi153.stdout: "client_max_inline_size": "4096", 2024-09-22T03:08:13.470 INFO:teuthology.orchestra.run.smithi153.stdout: "client_max_retries_on_remount_failure": "5", 2024-09-22T03:08:13.471 INFO:teuthology.orchestra.run.smithi153.stdout: "client_mds_namespace": "", 2024-09-22T03:08:13.471 INFO:teuthology.orchestra.run.smithi153.stdout: "client_metadata": "", 2024-09-22T03:08:13.471 INFO:teuthology.orchestra.run.smithi153.stdout: "client_mount_gid": "-1", 2024-09-22T03:08:13.471 INFO:teuthology.orchestra.run.smithi153.stdout: "client_mount_timeout": "300", 2024-09-22T03:08:13.471 INFO:teuthology.orchestra.run.smithi153.stdout: "client_mount_uid": "-1", 2024-09-22T03:08:13.471 INFO:teuthology.orchestra.run.smithi153.stdout: "client_mountpoint": "/", 2024-09-22T03:08:13.471 INFO:teuthology.orchestra.run.smithi153.stdout: "client_notify_timeout": "10", 2024-09-22T03:08:13.471 INFO:teuthology.orchestra.run.smithi153.stdout: "client_oc": "true", 2024-09-22T03:08:13.471 INFO:teuthology.orchestra.run.smithi153.stdout: "client_oc_max_dirty": "104857600", 2024-09-22T03:08:13.471 INFO:teuthology.orchestra.run.smithi153.stdout: "client_oc_max_dirty_age": "5.000000", 2024-09-22T03:08:13.471 INFO:teuthology.orchestra.run.smithi153.stdout: "client_oc_max_objects": "1000", 2024-09-22T03:08:13.471 INFO:teuthology.orchestra.run.smithi153.stdout: "client_oc_size": "209715200", 2024-09-22T03:08:13.471 INFO:teuthology.orchestra.run.smithi153.stdout: "client_oc_target_dirty": "8388608", 2024-09-22T03:08:13.471 INFO:teuthology.orchestra.run.smithi153.stdout: "client_permissions": "true", 2024-09-22T03:08:13.472 INFO:teuthology.orchestra.run.smithi153.stdout: "client_quota": "true", 2024-09-22T03:08:13.472 INFO:teuthology.orchestra.run.smithi153.stdout: "client_quota_df": "true", 2024-09-22T03:08:13.472 INFO:teuthology.orchestra.run.smithi153.stdout: "client_readahead_max_bytes": "0", 2024-09-22T03:08:13.472 INFO:teuthology.orchestra.run.smithi153.stdout: "client_readahead_max_periods": "4", 2024-09-22T03:08:13.472 INFO:teuthology.orchestra.run.smithi153.stdout: "client_readahead_min": "131072", 2024-09-22T03:08:13.472 INFO:teuthology.orchestra.run.smithi153.stdout: "client_reconnect_stale": "false", 2024-09-22T03:08:13.472 INFO:teuthology.orchestra.run.smithi153.stdout: "client_shutdown_timeout": "30", 2024-09-22T03:08:13.472 INFO:teuthology.orchestra.run.smithi153.stdout: "client_snapdir": ".snap", 2024-09-22T03:08:13.472 INFO:teuthology.orchestra.run.smithi153.stdout: "client_tick_interval": "1", 2024-09-22T03:08:13.472 INFO:teuthology.orchestra.run.smithi153.stdout: "client_trace": "", 2024-09-22T03:08:13.472 INFO:teuthology.orchestra.run.smithi153.stdout: "client_try_dentry_invalidate": "false", 2024-09-22T03:08:13.472 INFO:teuthology.orchestra.run.smithi153.stdout: "client_use_faked_inos": "false", 2024-09-22T03:08:13.472 INFO:teuthology.orchestra.run.smithi153.stdout: "client_use_random_mds": "false", 2024-09-22T03:08:13.472 INFO:teuthology.orchestra.run.smithi153.stdout: "clog_to_graylog": "false", 2024-09-22T03:08:13.473 INFO:teuthology.orchestra.run.smithi153.stdout: "clog_to_graylog_host": "127.0.0.1", 2024-09-22T03:08:13.473 INFO:teuthology.orchestra.run.smithi153.stdout: "clog_to_graylog_port": "12201", 2024-09-22T03:08:13.473 INFO:teuthology.orchestra.run.smithi153.stdout: "clog_to_monitors": "default=true", 2024-09-22T03:08:13.473 INFO:teuthology.orchestra.run.smithi153.stdout: "clog_to_syslog": "false", 2024-09-22T03:08:13.473 INFO:teuthology.orchestra.run.smithi153.stdout: "clog_to_syslog_facility": "default=daemon audit=local0", 2024-09-22T03:08:13.473 INFO:teuthology.orchestra.run.smithi153.stdout: "clog_to_syslog_level": "info", 2024-09-22T03:08:13.473 INFO:teuthology.orchestra.run.smithi153.stdout: "cluster_addr": "-", 2024-09-22T03:08:13.473 INFO:teuthology.orchestra.run.smithi153.stdout: "cluster_network": "", 2024-09-22T03:08:13.473 INFO:teuthology.orchestra.run.smithi153.stdout: "cluster_network_interface": "", 2024-09-22T03:08:13.473 INFO:teuthology.orchestra.run.smithi153.stdout: "compressor_zlib_isal": "false", 2024-09-22T03:08:13.473 INFO:teuthology.orchestra.run.smithi153.stdout: "compressor_zlib_level": "5", 2024-09-22T03:08:13.473 INFO:teuthology.orchestra.run.smithi153.stdout: "compressor_zlib_winsize": "-15", 2024-09-22T03:08:13.473 INFO:teuthology.orchestra.run.smithi153.stdout: "compressor_zstd_level": "1", 2024-09-22T03:08:13.473 INFO:teuthology.orchestra.run.smithi153.stdout: "container_image": "docker.io/ceph/daemon-base:latest-master-devel", 2024-09-22T03:08:13.473 INFO:teuthology.orchestra.run.smithi153.stdout: "crash_dir": "/var/lib/ceph/crash", 2024-09-22T03:08:13.474 INFO:teuthology.orchestra.run.smithi153.stdout: "crimson_alien_op_num_threads": "6", 2024-09-22T03:08:13.474 INFO:teuthology.orchestra.run.smithi153.stdout: "crimson_alien_thread_cpu_cores": "", 2024-09-22T03:08:13.474 INFO:teuthology.orchestra.run.smithi153.stdout: "crimson_osd_obc_lru_size": "512", 2024-09-22T03:08:13.474 INFO:teuthology.orchestra.run.smithi153.stdout: "crimson_osd_scheduler_concurrency": "0", 2024-09-22T03:08:13.474 INFO:teuthology.orchestra.run.smithi153.stdout: "crimson_osd_stat_interval": "0", 2024-09-22T03:08:13.474 INFO:teuthology.orchestra.run.smithi153.stdout: "crimson_seastar_cpu_cores": "", 2024-09-22T03:08:13.474 INFO:teuthology.orchestra.run.smithi153.stdout: "crimson_seastar_num_threads": "0", 2024-09-22T03:08:13.474 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "", 2024-09-22T03:08:13.474 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location_hook": "", 2024-09-22T03:08:13.474 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location_hook_timeout": "10", 2024-09-22T03:08:13.475 INFO:teuthology.orchestra.run.smithi153.stdout: "daemonize": "false", 2024-09-22T03:08:13.475 INFO:teuthology.orchestra.run.smithi153.stdout: "daos_pool": "tank", 2024-09-22T03:08:13.475 INFO:teuthology.orchestra.run.smithi153.stdout: "dbstore_config_uri": "file:/var/lib/ceph/radosgw/dbstore-config.db", 2024-09-22T03:08:13.475 INFO:teuthology.orchestra.run.smithi153.stdout: "dbstore_db_dir": "/var/lib/ceph/radosgw", 2024-09-22T03:08:13.475 INFO:teuthology.orchestra.run.smithi153.stdout: "dbstore_db_name_prefix": "dbstore", 2024-09-22T03:08:13.475 INFO:teuthology.orchestra.run.smithi153.stdout: "debug_alienstore": "0/5", 2024-09-22T03:08:13.475 INFO:teuthology.orchestra.run.smithi153.stdout: "debug_allow_any_pool_priority": "false", 2024-09-22T03:08:13.475 INFO:teuthology.orchestra.run.smithi153.stdout: "debug_asok": "1/5", 2024-09-22T03:08:13.475 INFO:teuthology.orchestra.run.smithi153.stdout: "debug_asok_assert_abort": "false", 2024-09-22T03:08:13.475 INFO:teuthology.orchestra.run.smithi153.stdout: "debug_asserts_on_shutdown": "true", 2024-09-22T03:08:13.475 INFO:teuthology.orchestra.run.smithi153.stdout: "debug_auth": "20/20", 2024-09-22T03:08:13.475 INFO:teuthology.orchestra.run.smithi153.stdout: "debug_bdev": "1/3", 2024-09-22T03:08:13.475 INFO:teuthology.orchestra.run.smithi153.stdout: "debug_bluefs": "1/5", 2024-09-22T03:08:13.476 INFO:teuthology.orchestra.run.smithi153.stdout: "debug_bluestore": "1/5", 2024-09-22T03:08:13.476 INFO:teuthology.orchestra.run.smithi153.stdout: "debug_buffer": "0/1", 2024-09-22T03:08:13.476 INFO:teuthology.orchestra.run.smithi153.stdout: "debug_ceph_exporter": "1/5", 2024-09-22T03:08:13.476 INFO:teuthology.orchestra.run.smithi153.stdout: "debug_cephfs_mirror": "0/5", 2024-09-22T03:08:13.476 INFO:teuthology.orchestra.run.smithi153.stdout: "debug_cephsqlite": "0/5", 2024-09-22T03:08:13.476 INFO:teuthology.orchestra.run.smithi153.stdout: "debug_client": "0/5", 2024-09-22T03:08:13.476 INFO:teuthology.orchestra.run.smithi153.stdout: "debug_compressor": "1/5", 2024-09-22T03:08:13.476 INFO:teuthology.orchestra.run.smithi153.stdout: "debug_context": "0/1", 2024-09-22T03:08:13.476 INFO:teuthology.orchestra.run.smithi153.stdout: "debug_crimson_interrupt": "0/5", 2024-09-22T03:08:13.476 INFO:teuthology.orchestra.run.smithi153.stdout: "debug_crush": "1/1", 2024-09-22T03:08:13.476 INFO:teuthology.orchestra.run.smithi153.stdout: "debug_crypto": "1/5", 2024-09-22T03:08:13.476 INFO:teuthology.orchestra.run.smithi153.stdout: "debug_cyanstore": "0/5", 2024-09-22T03:08:13.476 INFO:teuthology.orchestra.run.smithi153.stdout: "debug_deliberately_leak_memory": "false", 2024-09-22T03:08:13.476 INFO:teuthology.orchestra.run.smithi153.stdout: "debug_disable_randomized_ping": "false", 2024-09-22T03:08:13.476 INFO:teuthology.orchestra.run.smithi153.stdout: "debug_dpdk": "1/5", 2024-09-22T03:08:13.477 INFO:teuthology.orchestra.run.smithi153.stdout: "debug_eventtrace": "1/5", 2024-09-22T03:08:13.477 INFO:teuthology.orchestra.run.smithi153.stdout: "debug_filer": "0/1", 2024-09-22T03:08:13.477 INFO:teuthology.orchestra.run.smithi153.stdout: "debug_filestore": "1/3", 2024-09-22T03:08:13.477 INFO:teuthology.orchestra.run.smithi153.stdout: "debug_finisher": "1/1", 2024-09-22T03:08:13.477 INFO:teuthology.orchestra.run.smithi153.stdout: "debug_fuse": "1/5", 2024-09-22T03:08:13.477 INFO:teuthology.orchestra.run.smithi153.stdout: "debug_heartbeat_testing_span": "0", 2024-09-22T03:08:13.477 INFO:teuthology.orchestra.run.smithi153.stdout: "debug_heartbeatmap": "1/5", 2024-09-22T03:08:13.477 INFO:teuthology.orchestra.run.smithi153.stdout: "debug_immutable_obj_cache": "0/5", 2024-09-22T03:08:13.477 INFO:teuthology.orchestra.run.smithi153.stdout: "debug_javaclient": "1/5", 2024-09-22T03:08:13.477 INFO:teuthology.orchestra.run.smithi153.stdout: "debug_journal": "1/3", 2024-09-22T03:08:13.477 INFO:teuthology.orchestra.run.smithi153.stdout: "debug_journaler": "0/5", 2024-09-22T03:08:13.477 INFO:teuthology.orchestra.run.smithi153.stdout: "debug_kstore": "1/5", 2024-09-22T03:08:13.477 INFO:teuthology.orchestra.run.smithi153.stdout: "debug_lockdep": "0/1", 2024-09-22T03:08:13.477 INFO:teuthology.orchestra.run.smithi153.stdout: "debug_mclock": "1/5", 2024-09-22T03:08:13.478 INFO:teuthology.orchestra.run.smithi153.stdout: "debug_mds": "1/5", 2024-09-22T03:08:13.478 INFO:teuthology.orchestra.run.smithi153.stdout: "debug_mds_balancer": "1/5", 2024-09-22T03:08:13.478 INFO:teuthology.orchestra.run.smithi153.stdout: "debug_mds_locker": "1/5", 2024-09-22T03:08:13.478 INFO:teuthology.orchestra.run.smithi153.stdout: "debug_mds_log": "1/5", 2024-09-22T03:08:13.478 INFO:teuthology.orchestra.run.smithi153.stdout: "debug_mds_log_expire": "1/5", 2024-09-22T03:08:13.478 INFO:teuthology.orchestra.run.smithi153.stdout: "debug_mds_migrator": "1/5", 2024-09-22T03:08:13.478 INFO:teuthology.orchestra.run.smithi153.stdout: "debug_mds_quiesce": "3/5", 2024-09-22T03:08:13.478 INFO:teuthology.orchestra.run.smithi153.stdout: "debug_memstore": "1/5", 2024-09-22T03:08:13.478 INFO:teuthology.orchestra.run.smithi153.stdout: "debug_mgr": "2/5", 2024-09-22T03:08:13.478 INFO:teuthology.orchestra.run.smithi153.stdout: "debug_mgrc": "1/5", 2024-09-22T03:08:13.478 INFO:teuthology.orchestra.run.smithi153.stdout: "debug_mon": "20/20", 2024-09-22T03:08:13.478 INFO:teuthology.orchestra.run.smithi153.stdout: "debug_monc": "0/10", 2024-09-22T03:08:13.478 INFO:teuthology.orchestra.run.smithi153.stdout: "debug_ms": "1/1", 2024-09-22T03:08:13.479 INFO:teuthology.orchestra.run.smithi153.stdout: "debug_none": "0/5", 2024-09-22T03:08:13.479 INFO:teuthology.orchestra.run.smithi153.stdout: "debug_objclass": "0/5", 2024-09-22T03:08:13.479 INFO:teuthology.orchestra.run.smithi153.stdout: "debug_objectcacher": "0/5", 2024-09-22T03:08:13.479 INFO:teuthology.orchestra.run.smithi153.stdout: "debug_objecter": "0/1", 2024-09-22T03:08:13.479 INFO:teuthology.orchestra.run.smithi153.stdout: "debug_optracker": "0/5", 2024-09-22T03:08:13.479 INFO:teuthology.orchestra.run.smithi153.stdout: "debug_osd": "1/5", 2024-09-22T03:08:13.479 INFO:teuthology.orchestra.run.smithi153.stdout: "debug_paxos": "20/20", 2024-09-22T03:08:13.479 INFO:teuthology.orchestra.run.smithi153.stdout: "debug_perfcounter": "1/5", 2024-09-22T03:08:13.479 INFO:teuthology.orchestra.run.smithi153.stdout: "debug_prioritycache": "1/5", 2024-09-22T03:08:13.479 INFO:teuthology.orchestra.run.smithi153.stdout: "debug_rados": "0/5", 2024-09-22T03:08:13.479 INFO:teuthology.orchestra.run.smithi153.stdout: "debug_rbd": "0/5", 2024-09-22T03:08:13.480 INFO:teuthology.orchestra.run.smithi153.stdout: "debug_rbd_mirror": "0/5", 2024-09-22T03:08:13.480 INFO:teuthology.orchestra.run.smithi153.stdout: "debug_rbd_pwl": "0/5", 2024-09-22T03:08:13.480 INFO:teuthology.orchestra.run.smithi153.stdout: "debug_rbd_replay": "0/5", 2024-09-22T03:08:13.480 INFO:teuthology.orchestra.run.smithi153.stdout: "debug_refs": "0/0", 2024-09-22T03:08:13.480 INFO:teuthology.orchestra.run.smithi153.stdout: "debug_reserver": "1/1", 2024-09-22T03:08:13.480 INFO:teuthology.orchestra.run.smithi153.stdout: "debug_rgw": "1/5", 2024-09-22T03:08:13.480 INFO:teuthology.orchestra.run.smithi153.stdout: "debug_rgw_access": "1/5", 2024-09-22T03:08:13.480 INFO:teuthology.orchestra.run.smithi153.stdout: "debug_rgw_datacache": "1/5", 2024-09-22T03:08:13.480 INFO:teuthology.orchestra.run.smithi153.stdout: "debug_rgw_dbstore": "1/5", 2024-09-22T03:08:13.480 INFO:teuthology.orchestra.run.smithi153.stdout: "debug_rgw_flight": "1/5", 2024-09-22T03:08:13.480 INFO:teuthology.orchestra.run.smithi153.stdout: "debug_rgw_lifecycle": "1/5", 2024-09-22T03:08:13.480 INFO:teuthology.orchestra.run.smithi153.stdout: "debug_rgw_sync": "1/5", 2024-09-22T03:08:13.481 INFO:teuthology.orchestra.run.smithi153.stdout: "debug_rocksdb": "4/5", 2024-09-22T03:08:13.481 INFO:teuthology.orchestra.run.smithi153.stdout: "debug_seastore": "0/5", 2024-09-22T03:08:13.481 INFO:teuthology.orchestra.run.smithi153.stdout: "debug_seastore_backref": "0/5", 2024-09-22T03:08:13.481 INFO:teuthology.orchestra.run.smithi153.stdout: "debug_seastore_cache": "0/5", 2024-09-22T03:08:13.481 INFO:teuthology.orchestra.run.smithi153.stdout: "debug_seastore_cleaner": "0/5", 2024-09-22T03:08:13.481 INFO:teuthology.orchestra.run.smithi153.stdout: "debug_seastore_device": "0/5", 2024-09-22T03:08:13.481 INFO:teuthology.orchestra.run.smithi153.stdout: "debug_seastore_epm": "0/5", 2024-09-22T03:08:13.481 INFO:teuthology.orchestra.run.smithi153.stdout: "debug_seastore_fixedkv_tree": "0/5", 2024-09-22T03:08:13.481 INFO:teuthology.orchestra.run.smithi153.stdout: "debug_seastore_journal": "0/5", 2024-09-22T03:08:13.481 INFO:teuthology.orchestra.run.smithi153.stdout: "debug_seastore_lba": "0/5", 2024-09-22T03:08:13.481 INFO:teuthology.orchestra.run.smithi153.stdout: "debug_seastore_odata": "0/5", 2024-09-22T03:08:13.481 INFO:teuthology.orchestra.run.smithi153.stdout: "debug_seastore_omap": "0/5", 2024-09-22T03:08:13.482 INFO:teuthology.orchestra.run.smithi153.stdout: "debug_seastore_onode": "0/5", 2024-09-22T03:08:13.482 INFO:teuthology.orchestra.run.smithi153.stdout: "debug_seastore_t": "0/5", 2024-09-22T03:08:13.482 INFO:teuthology.orchestra.run.smithi153.stdout: "debug_seastore_tm": "0/5", 2024-09-22T03:08:13.482 INFO:teuthology.orchestra.run.smithi153.stdout: "debug_striper": "0/1", 2024-09-22T03:08:13.482 INFO:teuthology.orchestra.run.smithi153.stdout: "debug_test": "0/5", 2024-09-22T03:08:13.482 INFO:teuthology.orchestra.run.smithi153.stdout: "debug_throttle": "1/1", 2024-09-22T03:08:13.482 INFO:teuthology.orchestra.run.smithi153.stdout: "debug_timer": "0/1", 2024-09-22T03:08:13.482 INFO:teuthology.orchestra.run.smithi153.stdout: "debug_tp": "0/5", 2024-09-22T03:08:13.482 INFO:teuthology.orchestra.run.smithi153.stdout: "debug_trace": "1/5", 2024-09-22T03:08:13.482 INFO:teuthology.orchestra.run.smithi153.stdout: "defer_client_eviction_on_laggy_osds": "false", 2024-09-22T03:08:13.482 INFO:teuthology.orchestra.run.smithi153.stdout: "device_failure_prediction_mode": "none", 2024-09-22T03:08:13.482 INFO:teuthology.orchestra.run.smithi153.stdout: "enable_experimental_unrecoverable_data_corrupting_features": "crimson", 2024-09-22T03:08:13.483 INFO:teuthology.orchestra.run.smithi153.stdout: "erasure_code_dir": "/usr/lib64/ceph/erasure-code", 2024-09-22T03:08:13.483 INFO:teuthology.orchestra.run.smithi153.stdout: "err_to_graylog": "false", 2024-09-22T03:08:13.483 INFO:teuthology.orchestra.run.smithi153.stdout: "err_to_journald": "false", 2024-09-22T03:08:13.483 INFO:teuthology.orchestra.run.smithi153.stdout: "err_to_stderr": "true", 2024-09-22T03:08:13.483 INFO:teuthology.orchestra.run.smithi153.stdout: "err_to_syslog": "false", 2024-09-22T03:08:13.483 INFO:teuthology.orchestra.run.smithi153.stdout: "event_tracing": "false", 2024-09-22T03:08:13.483 INFO:teuthology.orchestra.run.smithi153.stdout: "exporter_addr": "0.0.0.0", 2024-09-22T03:08:13.483 INFO:teuthology.orchestra.run.smithi153.stdout: "exporter_http_port": "9926", 2024-09-22T03:08:13.483 INFO:teuthology.orchestra.run.smithi153.stdout: "exporter_prio_limit": "5", 2024-09-22T03:08:13.483 INFO:teuthology.orchestra.run.smithi153.stdout: "exporter_sock_dir": "/var/run/ceph/", 2024-09-22T03:08:13.483 INFO:teuthology.orchestra.run.smithi153.stdout: "exporter_sort_metrics": "true", 2024-09-22T03:08:13.483 INFO:teuthology.orchestra.run.smithi153.stdout: "exporter_stats_period": "5", 2024-09-22T03:08:13.484 INFO:teuthology.orchestra.run.smithi153.stdout: "fake_statfs_for_testing": "0", 2024-09-22T03:08:13.484 INFO:teuthology.orchestra.run.smithi153.stdout: "fatal_signal_handlers": "true", 2024-09-22T03:08:13.484 INFO:teuthology.orchestra.run.smithi153.stdout: "filer_max_purge_ops": "10", 2024-09-22T03:08:13.484 INFO:teuthology.orchestra.run.smithi153.stdout: "filer_max_truncate_ops": "128", 2024-09-22T03:08:13.484 INFO:teuthology.orchestra.run.smithi153.stdout: "filestore_apply_finisher_threads": "1", 2024-09-22T03:08:13.484 INFO:teuthology.orchestra.run.smithi153.stdout: "filestore_blackhole": "false", 2024-09-22T03:08:13.484 INFO:teuthology.orchestra.run.smithi153.stdout: "filestore_btrfs_clone_range": "true", 2024-09-22T03:08:13.484 INFO:teuthology.orchestra.run.smithi153.stdout: "filestore_btrfs_snap": "true", 2024-09-22T03:08:13.484 INFO:teuthology.orchestra.run.smithi153.stdout: "filestore_caller_concurrency": "10", 2024-09-22T03:08:13.484 INFO:teuthology.orchestra.run.smithi153.stdout: "filestore_collect_device_partition_information": "true", 2024-09-22T03:08:13.484 INFO:teuthology.orchestra.run.smithi153.stdout: "filestore_commit_timeout": "600.000000", 2024-09-22T03:08:13.484 INFO:teuthology.orchestra.run.smithi153.stdout: "filestore_debug_inject_read_err": "false", 2024-09-22T03:08:13.484 INFO:teuthology.orchestra.run.smithi153.stdout: "filestore_debug_omap_check": "false", 2024-09-22T03:08:13.484 INFO:teuthology.orchestra.run.smithi153.stdout: "filestore_debug_random_read_err": "0.000000", 2024-09-22T03:08:13.485 INFO:teuthology.orchestra.run.smithi153.stdout: "filestore_debug_verify_split": "false", 2024-09-22T03:08:13.485 INFO:teuthology.orchestra.run.smithi153.stdout: "filestore_dump_file": "", 2024-09-22T03:08:13.485 INFO:teuthology.orchestra.run.smithi153.stdout: "filestore_expected_throughput_bytes": "209715200.000000", 2024-09-22T03:08:13.485 INFO:teuthology.orchestra.run.smithi153.stdout: "filestore_expected_throughput_ops": "200.000000", 2024-09-22T03:08:13.485 INFO:teuthology.orchestra.run.smithi153.stdout: "filestore_fadvise": "true", 2024-09-22T03:08:13.485 INFO:teuthology.orchestra.run.smithi153.stdout: "filestore_fail_eio": "true", 2024-09-22T03:08:13.485 INFO:teuthology.orchestra.run.smithi153.stdout: "filestore_fd_cache_shards": "16", 2024-09-22T03:08:13.485 INFO:teuthology.orchestra.run.smithi153.stdout: "filestore_fd_cache_size": "128", 2024-09-22T03:08:13.485 INFO:teuthology.orchestra.run.smithi153.stdout: "filestore_fiemap": "false", 2024-09-22T03:08:13.485 INFO:teuthology.orchestra.run.smithi153.stdout: "filestore_fiemap_threshold": "4096", 2024-09-22T03:08:13.485 INFO:teuthology.orchestra.run.smithi153.stdout: "filestore_fsync_flushes_journal_data": "false", 2024-09-22T03:08:13.485 INFO:teuthology.orchestra.run.smithi153.stdout: "filestore_index_retry_probability": "0.000000", 2024-09-22T03:08:13.485 INFO:teuthology.orchestra.run.smithi153.stdout: "filestore_inject_stall": "0", 2024-09-22T03:08:13.485 INFO:teuthology.orchestra.run.smithi153.stdout: "filestore_journal_parallel": "false", 2024-09-22T03:08:13.486 INFO:teuthology.orchestra.run.smithi153.stdout: "filestore_journal_trailing": "false", 2024-09-22T03:08:13.486 INFO:teuthology.orchestra.run.smithi153.stdout: "filestore_journal_writeahead": "false", 2024-09-22T03:08:13.486 INFO:teuthology.orchestra.run.smithi153.stdout: "filestore_kill_at": "0", 2024-09-22T03:08:13.486 INFO:teuthology.orchestra.run.smithi153.stdout: "filestore_max_alloc_hint_size": "1048576", 2024-09-22T03:08:13.486 INFO:teuthology.orchestra.run.smithi153.stdout: "filestore_max_inline_xattr_size": "0", 2024-09-22T03:08:13.486 INFO:teuthology.orchestra.run.smithi153.stdout: "filestore_max_inline_xattr_size_btrfs": "2048", 2024-09-22T03:08:13.486 INFO:teuthology.orchestra.run.smithi153.stdout: "filestore_max_inline_xattr_size_other": "512", 2024-09-22T03:08:13.486 INFO:teuthology.orchestra.run.smithi153.stdout: "filestore_max_inline_xattr_size_xfs": "65536", 2024-09-22T03:08:13.486 INFO:teuthology.orchestra.run.smithi153.stdout: "filestore_max_inline_xattrs": "0", 2024-09-22T03:08:13.486 INFO:teuthology.orchestra.run.smithi153.stdout: "filestore_max_inline_xattrs_btrfs": "10", 2024-09-22T03:08:13.486 INFO:teuthology.orchestra.run.smithi153.stdout: "filestore_max_inline_xattrs_other": "2", 2024-09-22T03:08:13.486 INFO:teuthology.orchestra.run.smithi153.stdout: "filestore_max_inline_xattrs_xfs": "10", 2024-09-22T03:08:13.486 INFO:teuthology.orchestra.run.smithi153.stdout: "filestore_max_sync_interval": "5.000000", 2024-09-22T03:08:13.487 INFO:teuthology.orchestra.run.smithi153.stdout: "filestore_max_xattr_value_size": "0", 2024-09-22T03:08:13.487 INFO:teuthology.orchestra.run.smithi153.stdout: "filestore_max_xattr_value_size_btrfs": "65536", 2024-09-22T03:08:13.487 INFO:teuthology.orchestra.run.smithi153.stdout: "filestore_max_xattr_value_size_other": "1024", 2024-09-22T03:08:13.487 INFO:teuthology.orchestra.run.smithi153.stdout: "filestore_max_xattr_value_size_xfs": "65536", 2024-09-22T03:08:13.487 INFO:teuthology.orchestra.run.smithi153.stdout: "filestore_merge_threshold": "-10", 2024-09-22T03:08:13.487 INFO:teuthology.orchestra.run.smithi153.stdout: "filestore_min_sync_interval": "0.010000", 2024-09-22T03:08:13.487 INFO:teuthology.orchestra.run.smithi153.stdout: "filestore_odsync_write": "false", 2024-09-22T03:08:13.487 INFO:teuthology.orchestra.run.smithi153.stdout: "filestore_omap_backend": "rocksdb", 2024-09-22T03:08:13.487 INFO:teuthology.orchestra.run.smithi153.stdout: "filestore_omap_backend_path": "", 2024-09-22T03:08:13.487 INFO:teuthology.orchestra.run.smithi153.stdout: "filestore_omap_header_cache_size": "1024", 2024-09-22T03:08:13.487 INFO:teuthology.orchestra.run.smithi153.stdout: "filestore_ondisk_finisher_threads": "1", 2024-09-22T03:08:13.487 INFO:teuthology.orchestra.run.smithi153.stdout: "filestore_op_thread_suicide_timeout": "180", 2024-09-22T03:08:13.487 INFO:teuthology.orchestra.run.smithi153.stdout: "filestore_op_thread_timeout": "60", 2024-09-22T03:08:13.488 INFO:teuthology.orchestra.run.smithi153.stdout: "filestore_op_threads": "2", 2024-09-22T03:08:13.488 INFO:teuthology.orchestra.run.smithi153.stdout: "filestore_punch_hole": "false", 2024-09-22T03:08:13.488 INFO:teuthology.orchestra.run.smithi153.stdout: "filestore_queue_high_delay_multiple": "0.000000", 2024-09-22T03:08:13.488 INFO:teuthology.orchestra.run.smithi153.stdout: "filestore_queue_high_delay_multiple_bytes": "0.000000", 2024-09-22T03:08:13.488 INFO:teuthology.orchestra.run.smithi153.stdout: "filestore_queue_high_delay_multiple_ops": "0.000000", 2024-09-22T03:08:13.488 INFO:teuthology.orchestra.run.smithi153.stdout: "filestore_queue_high_threshhold": "0.900000", 2024-09-22T03:08:13.488 INFO:teuthology.orchestra.run.smithi153.stdout: "filestore_queue_low_threshhold": "0.300000", 2024-09-22T03:08:13.488 INFO:teuthology.orchestra.run.smithi153.stdout: "filestore_queue_max_bytes": "104857600", 2024-09-22T03:08:13.488 INFO:teuthology.orchestra.run.smithi153.stdout: "filestore_queue_max_delay_multiple": "0.000000", 2024-09-22T03:08:13.488 INFO:teuthology.orchestra.run.smithi153.stdout: "filestore_queue_max_delay_multiple_bytes": "0.000000", 2024-09-22T03:08:13.488 INFO:teuthology.orchestra.run.smithi153.stdout: "filestore_queue_max_delay_multiple_ops": "0.000000", 2024-09-22T03:08:13.488 INFO:teuthology.orchestra.run.smithi153.stdout: "filestore_queue_max_ops": "50", 2024-09-22T03:08:13.488 INFO:teuthology.orchestra.run.smithi153.stdout: "filestore_rocksdb_options": "max_background_jobs=10,compaction_readahead_size=2097152,compression=kNoCompression", 2024-09-22T03:08:13.489 INFO:teuthology.orchestra.run.smithi153.stdout: "filestore_seek_data_hole": "false", 2024-09-22T03:08:13.489 INFO:teuthology.orchestra.run.smithi153.stdout: "filestore_sloppy_crc": "false", 2024-09-22T03:08:13.489 INFO:teuthology.orchestra.run.smithi153.stdout: "filestore_sloppy_crc_block_size": "65536", 2024-09-22T03:08:13.489 INFO:teuthology.orchestra.run.smithi153.stdout: "filestore_splice": "false", 2024-09-22T03:08:13.489 INFO:teuthology.orchestra.run.smithi153.stdout: "filestore_split_multiple": "2", 2024-09-22T03:08:13.489 INFO:teuthology.orchestra.run.smithi153.stdout: "filestore_split_rand_factor": "20", 2024-09-22T03:08:13.489 INFO:teuthology.orchestra.run.smithi153.stdout: "filestore_update_to": "1000", 2024-09-22T03:08:13.489 INFO:teuthology.orchestra.run.smithi153.stdout: "filestore_wbthrottle_btrfs_bytes_hard_limit": "419430400", 2024-09-22T03:08:13.489 INFO:teuthology.orchestra.run.smithi153.stdout: "filestore_wbthrottle_btrfs_bytes_start_flusher": "41943040", 2024-09-22T03:08:13.489 INFO:teuthology.orchestra.run.smithi153.stdout: "filestore_wbthrottle_btrfs_inodes_hard_limit": "5000", 2024-09-22T03:08:13.489 INFO:teuthology.orchestra.run.smithi153.stdout: "filestore_wbthrottle_btrfs_inodes_start_flusher": "500", 2024-09-22T03:08:13.489 INFO:teuthology.orchestra.run.smithi153.stdout: "filestore_wbthrottle_btrfs_ios_hard_limit": "5000", 2024-09-22T03:08:13.489 INFO:teuthology.orchestra.run.smithi153.stdout: "filestore_wbthrottle_btrfs_ios_start_flusher": "500", 2024-09-22T03:08:13.489 INFO:teuthology.orchestra.run.smithi153.stdout: "filestore_wbthrottle_enable": "true", 2024-09-22T03:08:13.490 INFO:teuthology.orchestra.run.smithi153.stdout: "filestore_wbthrottle_xfs_bytes_hard_limit": "419430400", 2024-09-22T03:08:13.490 INFO:teuthology.orchestra.run.smithi153.stdout: "filestore_wbthrottle_xfs_bytes_start_flusher": "41943040", 2024-09-22T03:08:13.490 INFO:teuthology.orchestra.run.smithi153.stdout: "filestore_wbthrottle_xfs_inodes_hard_limit": "5000", 2024-09-22T03:08:13.490 INFO:teuthology.orchestra.run.smithi153.stdout: "filestore_wbthrottle_xfs_inodes_start_flusher": "500", 2024-09-22T03:08:13.490 INFO:teuthology.orchestra.run.smithi153.stdout: "filestore_wbthrottle_xfs_ios_hard_limit": "5000", 2024-09-22T03:08:13.490 INFO:teuthology.orchestra.run.smithi153.stdout: "filestore_wbthrottle_xfs_ios_start_flusher": "500", 2024-09-22T03:08:13.490 INFO:teuthology.orchestra.run.smithi153.stdout: "filestore_xfs_extsize": "false", 2024-09-22T03:08:13.490 INFO:teuthology.orchestra.run.smithi153.stdout: "filestore_zfs_snap": "false", 2024-09-22T03:08:13.490 INFO:teuthology.orchestra.run.smithi153.stdout: "fio_dir": "/tmp/fio", 2024-09-22T03:08:13.490 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "f17abef2-bc08-46d2-abf1-d47a95f0e2ee", 2024-09-22T03:08:13.490 INFO:teuthology.orchestra.run.smithi153.stdout: "fuse_allow_other": "true", 2024-09-22T03:08:13.490 INFO:teuthology.orchestra.run.smithi153.stdout: "fuse_atomic_o_trunc": "true", 2024-09-22T03:08:13.490 INFO:teuthology.orchestra.run.smithi153.stdout: "fuse_big_writes": "true", 2024-09-22T03:08:13.491 INFO:teuthology.orchestra.run.smithi153.stdout: "fuse_debug": "false", 2024-09-22T03:08:13.491 INFO:teuthology.orchestra.run.smithi153.stdout: "fuse_default_permissions": "false", 2024-09-22T03:08:13.491 INFO:teuthology.orchestra.run.smithi153.stdout: "fuse_disable_pagecache": "false", 2024-09-22T03:08:13.491 INFO:teuthology.orchestra.run.smithi153.stdout: "fuse_max_write": "0", 2024-09-22T03:08:13.491 INFO:teuthology.orchestra.run.smithi153.stdout: "fuse_multithreaded": "true", 2024-09-22T03:08:13.491 INFO:teuthology.orchestra.run.smithi153.stdout: "fuse_require_active_mds": "true", 2024-09-22T03:08:13.491 INFO:teuthology.orchestra.run.smithi153.stdout: "fuse_set_user_groups": "true", 2024-09-22T03:08:13.491 INFO:teuthology.orchestra.run.smithi153.stdout: "fuse_splice_move": "true", 2024-09-22T03:08:13.491 INFO:teuthology.orchestra.run.smithi153.stdout: "fuse_splice_read": "true", 2024-09-22T03:08:13.491 INFO:teuthology.orchestra.run.smithi153.stdout: "fuse_splice_write": "true", 2024-09-22T03:08:13.491 INFO:teuthology.orchestra.run.smithi153.stdout: "fuse_syncfs_on_mksnap": "true", 2024-09-22T03:08:13.491 INFO:teuthology.orchestra.run.smithi153.stdout: "fuse_use_invalidate_cb": "true", 2024-09-22T03:08:13.491 INFO:teuthology.orchestra.run.smithi153.stdout: "gss_ktab_client_file": "/var/lib/ceph/mon.a/gss_client_mon.a.ktab", 2024-09-22T03:08:13.491 INFO:teuthology.orchestra.run.smithi153.stdout: "gss_target_name": "ceph", 2024-09-22T03:08:13.492 INFO:teuthology.orchestra.run.smithi153.stdout: "heartbeat_file": "", 2024-09-22T03:08:13.492 INFO:teuthology.orchestra.run.smithi153.stdout: "heartbeat_inject_failure": "0", 2024-09-22T03:08:13.492 INFO:teuthology.orchestra.run.smithi153.stdout: "heartbeat_interval": "5", 2024-09-22T03:08:13.492 INFO:teuthology.orchestra.run.smithi153.stdout: "host": "smithi153", 2024-09-22T03:08:13.492 INFO:teuthology.orchestra.run.smithi153.stdout: "immutable_object_cache_client_dedicated_thread_num": "2", 2024-09-22T03:08:13.492 INFO:teuthology.orchestra.run.smithi153.stdout: "immutable_object_cache_max_inflight_ops": "128", 2024-09-22T03:08:13.492 INFO:teuthology.orchestra.run.smithi153.stdout: "immutable_object_cache_max_size": "1073741824", 2024-09-22T03:08:13.492 INFO:teuthology.orchestra.run.smithi153.stdout: "immutable_object_cache_path": "/tmp/ceph_immutable_object_cache", 2024-09-22T03:08:13.492 INFO:teuthology.orchestra.run.smithi153.stdout: "immutable_object_cache_qos_bps_burst": "0", 2024-09-22T03:08:13.492 INFO:teuthology.orchestra.run.smithi153.stdout: "immutable_object_cache_qos_bps_burst_seconds": "1", 2024-09-22T03:08:13.492 INFO:teuthology.orchestra.run.smithi153.stdout: "immutable_object_cache_qos_bps_limit": "0", 2024-09-22T03:08:13.492 INFO:teuthology.orchestra.run.smithi153.stdout: "immutable_object_cache_qos_iops_burst": "0", 2024-09-22T03:08:13.492 INFO:teuthology.orchestra.run.smithi153.stdout: "immutable_object_cache_qos_iops_burst_seconds": "1", 2024-09-22T03:08:13.493 INFO:teuthology.orchestra.run.smithi153.stdout: "immutable_object_cache_qos_iops_limit": "0", 2024-09-22T03:08:13.493 INFO:teuthology.orchestra.run.smithi153.stdout: "immutable_object_cache_qos_schedule_tick_min": "50", 2024-09-22T03:08:13.493 INFO:teuthology.orchestra.run.smithi153.stdout: "immutable_object_cache_sock": "/var/run/ceph/immutable_object_cache_sock", 2024-09-22T03:08:13.493 INFO:teuthology.orchestra.run.smithi153.stdout: "immutable_object_cache_watermark": "0.900000", 2024-09-22T03:08:13.493 INFO:teuthology.orchestra.run.smithi153.stdout: "inject_early_sigterm": "false", 2024-09-22T03:08:13.493 INFO:teuthology.orchestra.run.smithi153.stdout: "jaeger_agent_port": "6799", 2024-09-22T03:08:13.493 INFO:teuthology.orchestra.run.smithi153.stdout: "jaeger_tracing_enable": "false", 2024-09-22T03:08:13.493 INFO:teuthology.orchestra.run.smithi153.stdout: "journal_aio": "true", 2024-09-22T03:08:13.493 INFO:teuthology.orchestra.run.smithi153.stdout: "journal_align_min_size": "65536", 2024-09-22T03:08:13.493 INFO:teuthology.orchestra.run.smithi153.stdout: "journal_block_align": "true", 2024-09-22T03:08:13.493 INFO:teuthology.orchestra.run.smithi153.stdout: "journal_block_size": "4096", 2024-09-22T03:08:13.493 INFO:teuthology.orchestra.run.smithi153.stdout: "journal_dio": "true", 2024-09-22T03:08:13.493 INFO:teuthology.orchestra.run.smithi153.stdout: "journal_discard": "false", 2024-09-22T03:08:13.493 INFO:teuthology.orchestra.run.smithi153.stdout: "journal_force_aio": "false", 2024-09-22T03:08:13.494 INFO:teuthology.orchestra.run.smithi153.stdout: "journal_ignore_corruption": "false", 2024-09-22T03:08:13.494 INFO:teuthology.orchestra.run.smithi153.stdout: "journal_max_write_bytes": "10485760", 2024-09-22T03:08:13.494 INFO:teuthology.orchestra.run.smithi153.stdout: "journal_max_write_entries": "100", 2024-09-22T03:08:13.494 INFO:teuthology.orchestra.run.smithi153.stdout: "journal_replay_from": "0", 2024-09-22T03:08:13.494 INFO:teuthology.orchestra.run.smithi153.stdout: "journal_throttle_high_multiple": "0.000000", 2024-09-22T03:08:13.494 INFO:teuthology.orchestra.run.smithi153.stdout: "journal_throttle_high_threshhold": "0.900000", 2024-09-22T03:08:13.494 INFO:teuthology.orchestra.run.smithi153.stdout: "journal_throttle_low_threshhold": "0.600000", 2024-09-22T03:08:13.494 INFO:teuthology.orchestra.run.smithi153.stdout: "journal_throttle_max_multiple": "0.000000", 2024-09-22T03:08:13.494 INFO:teuthology.orchestra.run.smithi153.stdout: "journal_write_header_frequency": "0", 2024-09-22T03:08:13.494 INFO:teuthology.orchestra.run.smithi153.stdout: "journal_zero_on_create": "false", 2024-09-22T03:08:13.494 INFO:teuthology.orchestra.run.smithi153.stdout: "journaler_prefetch_periods": "10", 2024-09-22T03:08:13.494 INFO:teuthology.orchestra.run.smithi153.stdout: "journaler_prezero_periods": "5", 2024-09-22T03:08:13.494 INFO:teuthology.orchestra.run.smithi153.stdout: "journaler_write_head_interval": "15", 2024-09-22T03:08:13.494 INFO:teuthology.orchestra.run.smithi153.stdout: "key": "", 2024-09-22T03:08:13.494 INFO:teuthology.orchestra.run.smithi153.stdout: "keyfile": "", 2024-09-22T03:08:13.495 INFO:teuthology.orchestra.run.smithi153.stdout: "keyring": "/var/lib/ceph/mon/ceph-a/keyring", 2024-09-22T03:08:13.495 INFO:teuthology.orchestra.run.smithi153.stdout: "kstore_backend": "rocksdb", 2024-09-22T03:08:13.495 INFO:teuthology.orchestra.run.smithi153.stdout: "kstore_default_stripe_size": "65536", 2024-09-22T03:08:13.495 INFO:teuthology.orchestra.run.smithi153.stdout: "kstore_fsck_on_mount": "false", 2024-09-22T03:08:13.495 INFO:teuthology.orchestra.run.smithi153.stdout: "kstore_fsck_on_mount_deep": "true", 2024-09-22T03:08:13.495 INFO:teuthology.orchestra.run.smithi153.stdout: "kstore_max_bytes": "67108864", 2024-09-22T03:08:13.495 INFO:teuthology.orchestra.run.smithi153.stdout: "kstore_max_ops": "512", 2024-09-22T03:08:13.495 INFO:teuthology.orchestra.run.smithi153.stdout: "kstore_nid_prealloc": "1024", 2024-09-22T03:08:13.495 INFO:teuthology.orchestra.run.smithi153.stdout: "kstore_onode_map_size": "1024", 2024-09-22T03:08:13.495 INFO:teuthology.orchestra.run.smithi153.stdout: "kstore_rocksdb_options": "compression=kNoCompression", 2024-09-22T03:08:13.495 INFO:teuthology.orchestra.run.smithi153.stdout: "kstore_sync_submit_transaction": "false", 2024-09-22T03:08:13.495 INFO:teuthology.orchestra.run.smithi153.stdout: "kstore_sync_transaction": "false", 2024-09-22T03:08:13.495 INFO:teuthology.orchestra.run.smithi153.stdout: "librados_thread_count": "2", 2024-09-22T03:08:13.495 INFO:teuthology.orchestra.run.smithi153.stdout: "lockdep": "false", 2024-09-22T03:08:13.496 INFO:teuthology.orchestra.run.smithi153.stdout: "lockdep_force_backtrace": "false", 2024-09-22T03:08:13.496 INFO:teuthology.orchestra.run.smithi153.stdout: "log_coarse_timestamps": "true", 2024-09-22T03:08:13.496 INFO:teuthology.orchestra.run.smithi153.stdout: "log_file": "/var/log/ceph/ceph-mon.a.log", 2024-09-22T03:08:13.496 INFO:teuthology.orchestra.run.smithi153.stdout: "log_flush_on_exit": "false", 2024-09-22T03:08:13.496 INFO:teuthology.orchestra.run.smithi153.stdout: "log_graylog_host": "127.0.0.1", 2024-09-22T03:08:13.496 INFO:teuthology.orchestra.run.smithi153.stdout: "log_graylog_port": "12201", 2024-09-22T03:08:13.496 INFO:teuthology.orchestra.run.smithi153.stdout: "log_max_new": "1000", 2024-09-22T03:08:13.496 INFO:teuthology.orchestra.run.smithi153.stdout: "log_max_recent": "10000", 2024-09-22T03:08:13.496 INFO:teuthology.orchestra.run.smithi153.stdout: "log_stderr_prefix": "", 2024-09-22T03:08:13.496 INFO:teuthology.orchestra.run.smithi153.stdout: "log_stop_at_utilization": "0.970000", 2024-09-22T03:08:13.496 INFO:teuthology.orchestra.run.smithi153.stdout: "log_to_file": "true", 2024-09-22T03:08:13.496 INFO:teuthology.orchestra.run.smithi153.stdout: "log_to_graylog": "false", 2024-09-22T03:08:13.496 INFO:teuthology.orchestra.run.smithi153.stdout: "log_to_journald": "false", 2024-09-22T03:08:13.497 INFO:teuthology.orchestra.run.smithi153.stdout: "log_to_stderr": "false", 2024-09-22T03:08:13.497 INFO:teuthology.orchestra.run.smithi153.stdout: "log_to_syslog": "false", 2024-09-22T03:08:13.497 INFO:teuthology.orchestra.run.smithi153.stdout: "max_rotating_auth_attempts": "10", 2024-09-22T03:08:13.497 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_abort_on_newly_corrupt_dentry": "true", 2024-09-22T03:08:13.497 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_action_on_write_error": "1", 2024-09-22T03:08:13.497 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_alternate_name_max": "8192", 2024-09-22T03:08:13.497 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_asio_thread_count": "2", 2024-09-22T03:08:13.497 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_bal_export_pin": "true", 2024-09-22T03:08:13.497 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_bal_fragment_dirs": "true", 2024-09-22T03:08:13.497 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_bal_fragment_fast_factor": "1.500000", 2024-09-22T03:08:13.497 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_bal_fragment_interval": "5", 2024-09-22T03:08:13.497 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_bal_fragment_size_max": "100000", 2024-09-22T03:08:13.497 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_bal_idle_threshold": "0.000000", 2024-09-22T03:08:13.497 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_bal_interval": "10", 2024-09-22T03:08:13.497 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_bal_max": "-1", 2024-09-22T03:08:13.498 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_bal_max_until": "-1", 2024-09-22T03:08:13.498 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_bal_merge_size": "50", 2024-09-22T03:08:13.498 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_bal_midchunk": "0.300000", 2024-09-22T03:08:13.498 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_bal_min_rebalance": "0.100000", 2024-09-22T03:08:13.498 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_bal_min_start": "0.200000", 2024-09-22T03:08:13.498 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_bal_minchunk": "0.001000", 2024-09-22T03:08:13.498 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_bal_mode": "0", 2024-09-22T03:08:13.498 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_bal_need_max": "1.200000", 2024-09-22T03:08:13.498 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_bal_need_min": "0.800000", 2024-09-22T03:08:13.498 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_bal_overload_epochs": "2", 2024-09-22T03:08:13.498 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_bal_replicate_threshold": "8000.000000", 2024-09-22T03:08:13.498 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_bal_sample_interval": "3.000000", 2024-09-22T03:08:13.498 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_bal_split_bits": "3", 2024-09-22T03:08:13.499 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_bal_split_rd": "25000.000000", 2024-09-22T03:08:13.499 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_bal_split_size": "10000", 2024-09-22T03:08:13.499 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_bal_split_wr": "10000.000000", 2024-09-22T03:08:13.499 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_bal_target_decay": "10.000000", 2024-09-22T03:08:13.499 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_bal_unreplicate_threshold": "0.000000", 2024-09-22T03:08:13.499 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_beacon_grace": "15.000000", 2024-09-22T03:08:13.499 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_beacon_interval": "4.000000", 2024-09-22T03:08:13.499 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_beacon_mon_down_grace": "60", 2024-09-22T03:08:13.499 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_cache_memory_limit": "4294967296", 2024-09-22T03:08:13.499 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_cache_mid": "0.700000", 2024-09-22T03:08:13.499 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_cache_quiesce_decay_rate": "1.000000", 2024-09-22T03:08:13.499 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_cache_quiesce_delay": "0", 2024-09-22T03:08:13.499 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_cache_quiesce_sleep": "200", 2024-09-22T03:08:13.499 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_cache_quiesce_splitauth": "true", 2024-09-22T03:08:13.499 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_cache_quiesce_threshold": "524288", 2024-09-22T03:08:13.500 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_cache_release_free_interval": "10", 2024-09-22T03:08:13.500 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_cache_reservation": "0.050000", 2024-09-22T03:08:13.500 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_cache_trim_decay_rate": "1.000000", 2024-09-22T03:08:13.500 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_cache_trim_interval": "1", 2024-09-22T03:08:13.500 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_cache_trim_threshold": "262144", 2024-09-22T03:08:13.500 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_cap_acquisition_throttle_retry_request_timeout": "0.500000", 2024-09-22T03:08:13.500 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_cap_revoke_eviction_timeout": "0.000000", 2024-09-22T03:08:13.500 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_client_delegate_inos_pct": "50", 2024-09-22T03:08:13.500 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_client_prealloc_inos": "1000", 2024-09-22T03:08:13.500 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_client_writeable_range_max_inc_objs": "1024", 2024-09-22T03:08:13.500 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_connect_bootstrapping": "false", 2024-09-22T03:08:13.500 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_damage_table_max_entries": "10000", 2024-09-22T03:08:13.500 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_data": "/var/lib/ceph/mds/ceph-a", 2024-09-22T03:08:13.500 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_debug_auth_pins": "false", 2024-09-22T03:08:13.500 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_debug_frag": "false", 2024-09-22T03:08:13.501 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_debug_scatterstat": "false", 2024-09-22T03:08:13.501 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_debug_subtrees": "false", 2024-09-22T03:08:13.501 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_decay_halflife": "5.000000", 2024-09-22T03:08:13.501 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_default_dir_hash": "2", 2024-09-22T03:08:13.501 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_defer_session_stale": "true", 2024-09-22T03:08:13.501 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_deny_all_reconnect": "false", 2024-09-22T03:08:13.501 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_dir_keys_per_op": "16384", 2024-09-22T03:08:13.501 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_dir_max_commit_size": "10", 2024-09-22T03:08:13.501 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_dir_max_entries": "0", 2024-09-22T03:08:13.501 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_dir_prefetch": "true", 2024-09-22T03:08:13.501 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_dirstat_min_interval": "1.000000", 2024-09-22T03:08:13.501 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_dump_cache_after_rejoin": "false", 2024-09-22T03:08:13.501 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_dump_cache_on_map": "false", 2024-09-22T03:08:13.501 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_dump_cache_threshold_file": "0", 2024-09-22T03:08:13.502 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_dump_cache_threshold_formatter": "1073741824", 2024-09-22T03:08:13.502 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_early_reply": "true", 2024-09-22T03:08:13.502 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_enable_op_tracker": "true", 2024-09-22T03:08:13.502 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_enforce_unique_name": "true", 2024-09-22T03:08:13.502 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_export_ephemeral_distributed": "true", 2024-09-22T03:08:13.502 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_export_ephemeral_distributed_factor": "2.000000", 2024-09-22T03:08:13.502 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_export_ephemeral_random": "true", 2024-09-22T03:08:13.502 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_export_ephemeral_random_max": "0.010000", 2024-09-22T03:08:13.502 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_extraordinary_events_dump_interval": "0", 2024-09-22T03:08:13.502 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_forward_all_requests_to_auth": "false", 2024-09-22T03:08:13.502 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_freeze_tree_timeout": "30.000000", 2024-09-22T03:08:13.502 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_fscrypt_last_block_max_size": "4096", 2024-09-22T03:08:13.502 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_go_bad_corrupt_dentry": "true", 2024-09-22T03:08:13.503 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_hack_allow_loading_invalid_metadata": "false", 2024-09-22T03:08:13.503 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_health_cache_threshold": "1.500000", 2024-09-22T03:08:13.503 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_health_summarize_threshold": "10", 2024-09-22T03:08:13.503 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_heartbeat_grace": "15.000000", 2024-09-22T03:08:13.503 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_heartbeat_reset_grace": "1000", 2024-09-22T03:08:13.503 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_inject_health_dummy": "false", 2024-09-22T03:08:13.503 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_inject_journal_corrupt_dentry_first": "0.000000", 2024-09-22T03:08:13.503 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_inject_migrator_session_race": "false", 2024-09-22T03:08:13.503 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_inject_rename_corrupt_dentry_first": "0.000000", 2024-09-22T03:08:13.503 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_inject_skip_replaying_inotable": "false", 2024-09-22T03:08:13.503 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_inject_traceless_reply_probability": "0.000000", 2024-09-22T03:08:13.503 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_join_fs": "", 2024-09-22T03:08:13.503 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_journal_format": "1", 2024-09-22T03:08:13.504 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_kill_after_journal_logs_flushed": "false", 2024-09-22T03:08:13.504 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_kill_create_at": "0", 2024-09-22T03:08:13.504 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_kill_export_at": "0", 2024-09-22T03:08:13.504 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_kill_import_at": "0", 2024-09-22T03:08:13.504 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_kill_journal_at": "0", 2024-09-22T03:08:13.504 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_kill_journal_expire_at": "0", 2024-09-22T03:08:13.504 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_kill_journal_replay_at": "0", 2024-09-22T03:08:13.504 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_kill_link_at": "0", 2024-09-22T03:08:13.504 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_kill_mdstable_at": "0", 2024-09-22T03:08:13.504 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_kill_openc_at": "0", 2024-09-22T03:08:13.504 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_kill_rename_at": "0", 2024-09-22T03:08:13.504 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_kill_shutdown_at": "0", 2024-09-22T03:08:13.504 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_log_event_large_threshold": "524288", 2024-09-22T03:08:13.505 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_log_events_per_segment": "1024", 2024-09-22T03:08:13.505 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_log_major_segment_event_ratio": "12", 2024-09-22T03:08:13.505 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_log_max_events": "-1", 2024-09-22T03:08:13.505 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_log_max_segments": "128", 2024-09-22T03:08:13.505 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_log_pause": "false", 2024-09-22T03:08:13.505 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_log_segment_size": "0", 2024-09-22T03:08:13.505 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_log_skip_corrupt_events": "false", 2024-09-22T03:08:13.505 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_log_skip_unbounded_events": "false", 2024-09-22T03:08:13.505 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_log_trim_decay_rate": "1.000000", 2024-09-22T03:08:13.505 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_log_trim_threshold": "128", 2024-09-22T03:08:13.505 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_log_trim_upkeep_interval": "1000", 2024-09-22T03:08:13.505 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_log_warn_factor": "2.000000", 2024-09-22T03:08:13.506 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_max_caps_per_client": "1048576", 2024-09-22T03:08:13.506 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_max_completed_flushes": "100000", 2024-09-22T03:08:13.506 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_max_completed_requests": "100000", 2024-09-22T03:08:13.506 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_max_export_size": "20971520", 2024-09-22T03:08:13.506 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_max_file_recover": "32", 2024-09-22T03:08:13.506 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_max_purge_files": "64", 2024-09-22T03:08:13.506 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_max_purge_ops": "8192", 2024-09-22T03:08:13.506 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_max_purge_ops_per_pg": "0.500000", 2024-09-22T03:08:13.506 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_max_scrub_ops_in_progress": "5", 2024-09-22T03:08:13.506 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_max_snaps_per_dir": "100", 2024-09-22T03:08:13.506 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_metrics_update_interval": "2", 2024-09-22T03:08:13.506 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_min_caps_per_client": "100", 2024-09-22T03:08:13.507 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_min_caps_working_set": "10000", 2024-09-22T03:08:13.507 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_mon_shutdown_timeout": "5.000000", 2024-09-22T03:08:13.507 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_numa_node": "-1", 2024-09-22T03:08:13.507 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_oft_prefetch_dirfrags": "false", 2024-09-22T03:08:13.507 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_op_complaint_time": "30.000000", 2024-09-22T03:08:13.507 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_op_history_duration": "600", 2024-09-22T03:08:13.507 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_op_history_size": "20", 2024-09-22T03:08:13.507 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_op_history_slow_op_size": "20", 2024-09-22T03:08:13.507 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_op_history_slow_op_threshold": "10", 2024-09-22T03:08:13.507 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_op_log_threshold": "5", 2024-09-22T03:08:13.507 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_ping_grace": "15", 2024-09-22T03:08:13.507 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_ping_interval": "5", 2024-09-22T03:08:13.507 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_purge_queue_busy_flush_period": "1.000000", 2024-09-22T03:08:13.507 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_recall_global_max_decay_threshold": "131072", 2024-09-22T03:08:13.508 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_recall_max_caps": "30000", 2024-09-22T03:08:13.508 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_recall_max_decay_rate": "1.500000", 2024-09-22T03:08:13.508 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_recall_max_decay_threshold": "131072", 2024-09-22T03:08:13.508 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_recall_warning_decay_rate": "60.000000", 2024-09-22T03:08:13.508 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_recall_warning_threshold": "262144", 2024-09-22T03:08:13.508 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_reconnect_timeout": "45.000000", 2024-09-22T03:08:13.508 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_replay_interval": "1.000000", 2024-09-22T03:08:13.508 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_replay_unsafe_with_closed_session": "false", 2024-09-22T03:08:13.508 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_request_load_average_decay_rate": "60.000000", 2024-09-22T03:08:13.508 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_root_ino_gid": "0", 2024-09-22T03:08:13.508 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_root_ino_uid": "0", 2024-09-22T03:08:13.508 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_scatter_nudge_interval": "5.000000", 2024-09-22T03:08:13.508 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_session_blocklist_on_evict": "true", 2024-09-22T03:08:13.508 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_session_blocklist_on_timeout": "true", 2024-09-22T03:08:13.509 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_session_cache_liveness_decay_rate": "300.000000", 2024-09-22T03:08:13.509 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_session_cache_liveness_magnitude": "10", 2024-09-22T03:08:13.509 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_session_cap_acquisition_decay_rate": "30.000000", 2024-09-22T03:08:13.509 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_session_cap_acquisition_throttle": "100000", 2024-09-22T03:08:13.509 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_session_max_caps_throttle_ratio": "1.100000", 2024-09-22T03:08:13.509 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_session_metadata_threshold": "16777216", 2024-09-22T03:08:13.509 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_sessionmap_keys_per_op": "1024", 2024-09-22T03:08:13.509 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_shutdown_check": "0", 2024-09-22T03:08:13.509 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_skip_ino": "0", 2024-09-22T03:08:13.509 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_sleep_rank_change": "0.000000", 2024-09-22T03:08:13.509 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_snap_max_uid": "4294967294", 2024-09-22T03:08:13.509 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_snap_min_uid": "0", 2024-09-22T03:08:13.509 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_snap_rstat": "false", 2024-09-22T03:08:13.509 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_standby_replay_damaged": "false", 2024-09-22T03:08:13.509 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_symlink_recovery": "true", 2024-09-22T03:08:13.510 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_task_status_update_interval": "2.000000", 2024-09-22T03:08:13.510 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_thrash_exports": "0", 2024-09-22T03:08:13.510 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_thrash_fragments": "0", 2024-09-22T03:08:13.510 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_tick_interval": "5.000000", 2024-09-22T03:08:13.510 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_valgrind_exit": "false", 2024-09-22T03:08:13.510 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_verify_backtrace": "1", 2024-09-22T03:08:13.510 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_verify_scatter": "false", 2024-09-22T03:08:13.510 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_wipe_ino_prealloc": "false", 2024-09-22T03:08:13.510 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_wipe_sessions": "false", 2024-09-22T03:08:13.510 INFO:teuthology.orchestra.run.smithi153.stdout: "mempool_debug": "false", 2024-09-22T03:08:13.510 INFO:teuthology.orchestra.run.smithi153.stdout: "memstore_debug_omit_block_device_write": "false", 2024-09-22T03:08:13.510 INFO:teuthology.orchestra.run.smithi153.stdout: "memstore_device_bytes": "1073741824", 2024-09-22T03:08:13.510 INFO:teuthology.orchestra.run.smithi153.stdout: "memstore_page_set": "false", 2024-09-22T03:08:13.510 INFO:teuthology.orchestra.run.smithi153.stdout: "memstore_page_size": "65536", 2024-09-22T03:08:13.511 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr_client_bytes": "134217728", 2024-09-22T03:08:13.511 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr_client_messages": "512", 2024-09-22T03:08:13.511 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr_client_service_daemon_unregister_timeout": "1.000000", 2024-09-22T03:08:13.511 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr_connect_retry_interval": "1.000000", 2024-09-22T03:08:13.511 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr_data": "/var/lib/ceph/mgr/ceph-a", 2024-09-22T03:08:13.511 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr_debug_aggressive_pg_num_changes": "false", 2024-09-22T03:08:13.511 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr_disabled_modules": "", 2024-09-22T03:08:13.511 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr_enable_op_tracker": "true", 2024-09-22T03:08:13.511 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr_initial_modules": "restful iostat nfs", 2024-09-22T03:08:13.511 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr_max_pg_num_change": "128", 2024-09-22T03:08:13.511 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr_mds_bytes": "134217728", 2024-09-22T03:08:13.511 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr_mds_messages": "128", 2024-09-22T03:08:13.511 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr_module_path": "/usr/share/ceph/mgr", 2024-09-22T03:08:13.511 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr_mon_bytes": "134217728", 2024-09-22T03:08:13.512 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr_mon_messages": "128", 2024-09-22T03:08:13.512 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr_num_op_tracker_shard": "32", 2024-09-22T03:08:13.512 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr_op_complaint_time": "30.000000", 2024-09-22T03:08:13.512 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr_op_history_duration": "600", 2024-09-22T03:08:13.512 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr_op_history_size": "20", 2024-09-22T03:08:13.512 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr_op_history_slow_op_size": "20", 2024-09-22T03:08:13.512 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr_op_history_slow_op_threshold": "10.000000", 2024-09-22T03:08:13.512 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr_op_log_threshold": "5", 2024-09-22T03:08:13.512 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr_osd_bytes": "536870912", 2024-09-22T03:08:13.512 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr_osd_messages": "8192", 2024-09-22T03:08:13.512 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr_pool": "true", 2024-09-22T03:08:13.512 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr_service_beacon_grace": "60.000000", 2024-09-22T03:08:13.512 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr_standby_modules": "true", 2024-09-22T03:08:13.513 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr_stats_period": "5", 2024-09-22T03:08:13.513 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr_stats_threshold": "5", 2024-09-22T03:08:13.513 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr_tick_period": "2", 2024-09-22T03:08:13.513 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr_ttl_cache_expire_seconds": "0", 2024-09-22T03:08:13.513 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_accept_timeout_factor": "2.000000", 2024-09-22T03:08:13.513 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_allow_pool_delete": "true", 2024-09-22T03:08:13.513 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_allow_pool_size_one": "true", 2024-09-22T03:08:13.513 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_auth_validate_all_caps": "true", 2024-09-22T03:08:13.513 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_cache_target_full_warn_ratio": "0.660000", 2024-09-22T03:08:13.513 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_clean_pg_upmaps_per_chunk": "256", 2024-09-22T03:08:13.513 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_client_bytes": "104857600", 2024-09-22T03:08:13.513 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_client_directed_command_retry": "2", 2024-09-22T03:08:13.514 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_client_hunt_interval": "3.000000", 2024-09-22T03:08:13.514 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_client_hunt_interval_backoff": "1.500000", 2024-09-22T03:08:13.514 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_client_hunt_interval_max_multiple": "10.000000", 2024-09-22T03:08:13.514 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_client_hunt_interval_min_multiple": "1.000000", 2024-09-22T03:08:13.514 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_client_hunt_parallel": "3", 2024-09-22T03:08:13.514 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_client_log_interval": "1.000000", 2024-09-22T03:08:13.514 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_client_max_log_entries_per_message": "1000", 2024-09-22T03:08:13.514 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_client_ping_interval": "10.000000", 2024-09-22T03:08:13.514 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_client_ping_timeout": "30.000000", 2024-09-22T03:08:13.514 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_client_target_rank": "-1", 2024-09-22T03:08:13.514 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_clock_drift_allowed": "1.000000", 2024-09-22T03:08:13.514 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_clock_drift_warn_backoff": "5.000000", 2024-09-22T03:08:13.515 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_cluster_log_file": "default=/var/log/ceph/ceph.$channel.log cluster=/var/log/ceph/ceph.log", 2024-09-22T03:08:13.515 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_cluster_log_file_level": "debug", 2024-09-22T03:08:13.515 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_cluster_log_to_file": "true", 2024-09-22T03:08:13.515 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_cluster_log_to_graylog": "false", 2024-09-22T03:08:13.515 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_cluster_log_to_graylog_host": "127.0.0.1", 2024-09-22T03:08:13.515 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_cluster_log_to_graylog_port": "12201", 2024-09-22T03:08:13.515 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_cluster_log_to_journald": "false", 2024-09-22T03:08:13.515 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_cluster_log_to_stderr": "false", 2024-09-22T03:08:13.515 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_cluster_log_to_syslog": "default=false", 2024-09-22T03:08:13.515 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_cluster_log_to_syslog_facility": "daemon", 2024-09-22T03:08:13.515 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_cluster_log_to_syslog_level": "info", 2024-09-22T03:08:13.516 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_compact_on_bootstrap": "false", 2024-09-22T03:08:13.516 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_compact_on_start": "false", 2024-09-22T03:08:13.516 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_compact_on_trim": "true", 2024-09-22T03:08:13.516 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_con_tracker_persist_interval": "10", 2024-09-22T03:08:13.516 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_con_tracker_score_halflife": "43200", 2024-09-22T03:08:13.516 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_config_key_max_entry_size": "65536", 2024-09-22T03:08:13.516 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_cpu_threads": "4", 2024-09-22T03:08:13.516 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_crush_min_required_version": "hammer", 2024-09-22T03:08:13.516 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_daemon_bytes": "419430400", 2024-09-22T03:08:13.516 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_data": "/var/lib/ceph/mon/ceph-a", 2024-09-22T03:08:13.516 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_data_avail_crit": "5", 2024-09-22T03:08:13.516 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_data_avail_warn": "5", 2024-09-22T03:08:13.517 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_data_size_warn": "16106127360", 2024-09-22T03:08:13.517 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_debug_block_osdmap_trim": "false", 2024-09-22T03:08:13.517 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_debug_deprecated_as_obsolete": "false", 2024-09-22T03:08:13.517 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_debug_dump_json": "false", 2024-09-22T03:08:13.517 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_debug_dump_location": "/var/log/ceph/ceph-mon.a.tdump", 2024-09-22T03:08:13.517 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_debug_dump_transactions": "false", 2024-09-22T03:08:13.517 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_debug_extra_checks": "false", 2024-09-22T03:08:13.517 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_debug_no_initial_persistent_features": "false", 2024-09-22T03:08:13.517 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_debug_no_require_bluestore_for_ec_overwrites": "false", 2024-09-22T03:08:13.517 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_debug_no_require_reef": "false", 2024-09-22T03:08:13.517 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_debug_no_require_squid": "false", 2024-09-22T03:08:13.517 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_debug_unsafe_allow_tier_with_nonempty_snaps": "false", 2024-09-22T03:08:13.518 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_delta_reset_interval": "10.000000", 2024-09-22T03:08:13.518 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_dns_srv_name": "ceph-mon", 2024-09-22T03:08:13.518 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_down_mkfs_grace": "120", 2024-09-22T03:08:13.518 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_down_uptime_grace": "60", 2024-09-22T03:08:13.518 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_election_default_strategy": "1", 2024-09-22T03:08:13.518 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_election_timeout": "5.000000", 2024-09-22T03:08:13.518 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_elector_ignore_propose_margin": "0.000500", 2024-09-22T03:08:13.518 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_elector_ping_divisor": "2", 2024-09-22T03:08:13.518 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_elector_ping_timeout": "2.000000", 2024-09-22T03:08:13.518 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_enable_op_tracker": "true", 2024-09-22T03:08:13.518 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_fake_pool_delete": "false", 2024-09-22T03:08:13.518 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_force_quorum_join": "false", 2024-09-22T03:08:13.518 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_fsmap_prune_threshold": "300", 2024-09-22T03:08:13.519 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_globalid_prealloc": "10000", 2024-09-22T03:08:13.519 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_health_detail_to_clog": "false", 2024-09-22T03:08:13.519 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_health_log_update_period": "5", 2024-09-22T03:08:13.519 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_health_max_detail": "50", 2024-09-22T03:08:13.519 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_health_to_clog": "true", 2024-09-22T03:08:13.519 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_health_to_clog_interval": "600", 2024-09-22T03:08:13.519 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_health_to_clog_tick_interval": "60.000000", 2024-09-22T03:08:13.519 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_host": "172.21.15.153", 2024-09-22T03:08:13.519 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_host_override": "", 2024-09-22T03:08:13.519 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_initial_members": "", 2024-09-22T03:08:13.519 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_inject_pg_merge_bounce_probability": "0.000000", 2024-09-22T03:08:13.519 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_inject_sync_get_chunk_delay": "0.000000", 2024-09-22T03:08:13.519 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_inject_transaction_delay_max": "10.000000", 2024-09-22T03:08:13.519 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_inject_transaction_delay_probability": "0.000000", 2024-09-22T03:08:13.520 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_keyvaluedb": "rocksdb", 2024-09-22T03:08:13.520 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_lease": "5.000000", 2024-09-22T03:08:13.520 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_lease_ack_timeout_factor": "2.000000", 2024-09-22T03:08:13.520 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_lease_renew_interval_factor": "0.600000", 2024-09-22T03:08:13.520 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_log_full_interval": "50", 2024-09-22T03:08:13.520 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_log_max": "10000", 2024-09-22T03:08:13.520 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_log_max_summary": "50", 2024-09-22T03:08:13.520 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_max_log_entries_per_event": "4096", 2024-09-22T03:08:13.520 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_max_log_epochs": "500", 2024-09-22T03:08:13.520 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_max_mdsmap_epochs": "500", 2024-09-22T03:08:13.520 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_max_mgrmap_epochs": "500", 2024-09-22T03:08:13.520 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_max_osd": "10000", 2024-09-22T03:08:13.521 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_max_pg_per_osd": "10000", 2024-09-22T03:08:13.521 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_max_pool_pg_num": "65536", 2024-09-22T03:08:13.521 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_max_snap_prune_per_epoch": "100", 2024-09-22T03:08:13.521 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_mds_blocklist_interval": "86400.000000", 2024-09-22T03:08:13.521 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_mds_force_trim_to": "0", 2024-09-22T03:08:13.521 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_mds_skip_sanity": "false", 2024-09-22T03:08:13.521 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_memory_autotune": "true", 2024-09-22T03:08:13.521 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_memory_target": "2147483648", 2024-09-22T03:08:13.521 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_mgr_beacon_grace": "30", 2024-09-22T03:08:13.521 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_mgr_blocklist_interval": "86400.000000", 2024-09-22T03:08:13.521 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_mgr_digest_period": "5", 2024-09-22T03:08:13.521 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_mgr_inactive_grace": "60", 2024-09-22T03:08:13.522 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_mgr_mkfs_grace": "240", 2024-09-22T03:08:13.522 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_mgr_proxy_client_bytes_ratio": "0.300000", 2024-09-22T03:08:13.522 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_min_osdmap_epochs": "50", 2024-09-22T03:08:13.522 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_op_complaint_time": "30", 2024-09-22T03:08:13.522 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_op_history_duration": "600", 2024-09-22T03:08:13.522 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_op_history_size": "20", 2024-09-22T03:08:13.522 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_op_history_slow_op_size": "20", 2024-09-22T03:08:13.522 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_op_history_slow_op_threshold": "10", 2024-09-22T03:08:13.522 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_op_log_threshold": "5", 2024-09-22T03:08:13.522 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_osd_adjust_down_out_interval": "true", 2024-09-22T03:08:13.522 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_osd_adjust_heartbeat_grace": "true", 2024-09-22T03:08:13.522 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_osd_auto_mark_auto_out_in": "true", 2024-09-22T03:08:13.522 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_osd_auto_mark_in": "false", 2024-09-22T03:08:13.523 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_osd_auto_mark_new_in": "true", 2024-09-22T03:08:13.523 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_osd_backfillfull_ratio": "0.900000", 2024-09-22T03:08:13.523 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_osd_blocklist_default_expire": "3600.000000", 2024-09-22T03:08:13.523 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_osd_cache_size": "500", 2024-09-22T03:08:13.523 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_osd_cache_size_min": "134217728", 2024-09-22T03:08:13.523 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_osd_crush_smoke_test": "true", 2024-09-22T03:08:13.523 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_osd_destroyed_out_interval": "600", 2024-09-22T03:08:13.523 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_osd_down_out_interval": "600", 2024-09-22T03:08:13.523 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_osd_down_out_subtree_limit": "rack", 2024-09-22T03:08:13.523 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_osd_err_op_age_ratio": "128.000000", 2024-09-22T03:08:13.523 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_osd_force_trim_to": "0", 2024-09-22T03:08:13.523 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_osd_full_ratio": "0.950000", 2024-09-22T03:08:13.523 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_osd_initial_require_min_compat_client": "luminous", 2024-09-22T03:08:13.524 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_osd_laggy_halflife": "3600", 2024-09-22T03:08:13.524 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_osd_laggy_max_interval": "300", 2024-09-22T03:08:13.524 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_osd_laggy_weight": "0.300000", 2024-09-22T03:08:13.524 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_osd_mapping_pgs_per_chunk": "4096", 2024-09-22T03:08:13.524 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_osd_max_creating_pgs": "1024", 2024-09-22T03:08:13.524 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_osd_max_initial_pgs": "1024", 2024-09-22T03:08:13.524 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_osd_min_down_reporters": "2", 2024-09-22T03:08:13.524 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_osd_min_in_ratio": "0.750000", 2024-09-22T03:08:13.524 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_osd_min_up_ratio": "0.300000", 2024-09-22T03:08:13.524 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_osd_nearfull_ratio": "0.850000", 2024-09-22T03:08:13.524 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_osd_prime_pg_temp": "true", 2024-09-22T03:08:13.524 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_osd_prime_pg_temp_max_estimate": "0.250000", 2024-09-22T03:08:13.524 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_osd_prime_pg_temp_max_time": "0.500000", 2024-09-22T03:08:13.524 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_osd_report_timeout": "900", 2024-09-22T03:08:13.525 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_osd_reporter_subtree_level": "osd", 2024-09-22T03:08:13.525 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_osd_snap_trim_queue_warn_on": "32768", 2024-09-22T03:08:13.525 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_osd_warn_num_repaired": "10", 2024-09-22T03:08:13.525 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_osd_warn_op_age": "32.000000", 2024-09-22T03:08:13.525 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_osdmap_full_prune_enabled": "true", 2024-09-22T03:08:13.525 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_osdmap_full_prune_interval": "2", 2024-09-22T03:08:13.525 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_osdmap_full_prune_min": "15", 2024-09-22T03:08:13.525 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_osdmap_full_prune_txsize": "2", 2024-09-22T03:08:13.525 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_pg_check_down_all_threshold": "0.500000", 2024-09-22T03:08:13.525 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_pg_stuck_threshold": "60", 2024-09-22T03:08:13.525 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_pg_warn_max_object_skew": "0.000000", 2024-09-22T03:08:13.525 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_pg_warn_min_objects": "10000", 2024-09-22T03:08:13.525 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_pg_warn_min_per_osd": "0", 2024-09-22T03:08:13.525 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_pg_warn_min_pool_objects": "1000", 2024-09-22T03:08:13.525 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_pool_quota_crit_threshold": "0", 2024-09-22T03:08:13.526 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_pool_quota_warn_threshold": "0", 2024-09-22T03:08:13.526 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_probe_timeout": "2.000000", 2024-09-22T03:08:13.526 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_reweight_max_change": "0.050000", 2024-09-22T03:08:13.526 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_reweight_max_osds": "4", 2024-09-22T03:08:13.526 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_reweight_min_bytes_per_osd": "10", 2024-09-22T03:08:13.526 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_reweight_min_pgs_per_osd": "4", 2024-09-22T03:08:13.526 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_rocksdb_options": "write_buffer_size=33554432,compression=kNoCompression,level_compaction_dynamic_level_bytes=true", 2024-09-22T03:08:13.526 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_scrub_inject_crc_mismatch": "0.000000", 2024-09-22T03:08:13.526 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_scrub_inject_missing_keys": "0.000000", 2024-09-22T03:08:13.526 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_scrub_interval": "86400", 2024-09-22T03:08:13.526 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_scrub_max_keys": "100", 2024-09-22T03:08:13.526 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_scrub_timeout": "300", 2024-09-22T03:08:13.526 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_session_timeout": "300", 2024-09-22T03:08:13.527 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_smart_report_timeout": "5", 2024-09-22T03:08:13.527 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_stat_smooth_intervals": "6", 2024-09-22T03:08:13.527 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_stretch_cluster_recovery_ratio": "0.600000", 2024-09-22T03:08:13.527 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_stretch_pool_min_size": "2", 2024-09-22T03:08:13.527 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_stretch_pool_size": "4", 2024-09-22T03:08:13.527 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_stretch_recovery_min_wait": "15.000000", 2024-09-22T03:08:13.527 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_subscribe_interval": "86400.000000", 2024-09-22T03:08:13.527 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_sync_debug": "false", 2024-09-22T03:08:13.527 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_sync_max_payload_keys": "2000", 2024-09-22T03:08:13.527 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_sync_max_payload_size": "1048576", 2024-09-22T03:08:13.527 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_sync_provider_kill_at": "0", 2024-09-22T03:08:13.527 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_sync_requester_kill_at": "0", 2024-09-22T03:08:13.527 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_sync_timeout": "60.000000", 2024-09-22T03:08:13.527 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_target_pg_per_osd": "100", 2024-09-22T03:08:13.528 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_tick_interval": "5", 2024-09-22T03:08:13.528 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_timecheck_interval": "300.000000", 2024-09-22T03:08:13.528 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_timecheck_skew_interval": "30.000000", 2024-09-22T03:08:13.528 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_use_min_delay_socket": "false", 2024-09-22T03:08:13.528 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_warn_older_version_delay": "604800", 2024-09-22T03:08:13.528 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_warn_on_cache_pools_without_hit_sets": "true", 2024-09-22T03:08:13.528 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_warn_on_crush_straw_calc_version_zero": "false", 2024-09-22T03:08:13.528 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_warn_on_degraded_stretch_mode": "true", 2024-09-22T03:08:13.528 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_warn_on_filestore_osds": "false", 2024-09-22T03:08:13.528 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_warn_on_insecure_global_id_reclaim": "false", 2024-09-22T03:08:13.528 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_warn_on_insecure_global_id_reclaim_allowed": "false", 2024-09-22T03:08:13.528 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_warn_on_legacy_crush_tunables": "false", 2024-09-22T03:08:13.528 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_warn_on_misplaced": "false", 2024-09-22T03:08:13.529 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_warn_on_msgr2_not_enabled": "true", 2024-09-22T03:08:13.529 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_warn_on_older_version": "true", 2024-09-22T03:08:13.529 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_warn_on_osd_down_out_interval_zero": "false", 2024-09-22T03:08:13.529 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_warn_on_pool_no_app": "true", 2024-09-22T03:08:13.529 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_warn_on_pool_no_redundancy": "false", 2024-09-22T03:08:13.529 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_warn_on_pool_pg_num_not_power_of_two": "false", 2024-09-22T03:08:13.529 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_warn_on_slow_ping_ratio": "0.050000", 2024-09-22T03:08:13.529 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_warn_on_slow_ping_time": "0.000000", 2024-09-22T03:08:13.529 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_warn_on_too_few_osds": "false", 2024-09-22T03:08:13.529 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_warn_pg_not_deep_scrubbed_ratio": "0.750000", 2024-09-22T03:08:13.529 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_warn_pg_not_scrubbed_ratio": "0.500000", 2024-09-22T03:08:13.529 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": "", 2024-09-22T03:08:13.529 INFO:teuthology.orchestra.run.smithi153.stdout: "motr_admin_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-09-22T03:08:13.529 INFO:teuthology.orchestra.run.smithi153.stdout: "motr_admin_fid": "0x7200000000000001:0x0", 2024-09-22T03:08:13.530 INFO:teuthology.orchestra.run.smithi153.stdout: "motr_ha_endpoint": "192.168.180.182@tcp:12345:1:1", 2024-09-22T03:08:13.530 INFO:teuthology.orchestra.run.smithi153.stdout: "motr_my_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-09-22T03:08:13.530 INFO:teuthology.orchestra.run.smithi153.stdout: "motr_my_fid": "0x7200000000000001:0x0", 2024-09-22T03:08:13.530 INFO:teuthology.orchestra.run.smithi153.stdout: "motr_profile_fid": "0x7000000000000001:0x0", 2024-09-22T03:08:13.530 INFO:teuthology.orchestra.run.smithi153.stdout: "motr_tracing_enabled": "false", 2024-09-22T03:08:13.530 INFO:teuthology.orchestra.run.smithi153.stdout: "ms_async_op_threads": "3", 2024-09-22T03:08:13.530 INFO:teuthology.orchestra.run.smithi153.stdout: "ms_async_rdma_buffer_size": "131072", 2024-09-22T03:08:13.530 INFO:teuthology.orchestra.run.smithi153.stdout: "ms_async_rdma_cm": "false", 2024-09-22T03:08:13.530 INFO:teuthology.orchestra.run.smithi153.stdout: "ms_async_rdma_device_name": "", 2024-09-22T03:08:13.530 INFO:teuthology.orchestra.run.smithi153.stdout: "ms_async_rdma_dscp": "96", 2024-09-22T03:08:13.530 INFO:teuthology.orchestra.run.smithi153.stdout: "ms_async_rdma_enable_hugepage": "false", 2024-09-22T03:08:13.530 INFO:teuthology.orchestra.run.smithi153.stdout: "ms_async_rdma_gid_idx": "0", 2024-09-22T03:08:13.530 INFO:teuthology.orchestra.run.smithi153.stdout: "ms_async_rdma_local_gid": "", 2024-09-22T03:08:13.531 INFO:teuthology.orchestra.run.smithi153.stdout: "ms_async_rdma_polling_us": "1000", 2024-09-22T03:08:13.531 INFO:teuthology.orchestra.run.smithi153.stdout: "ms_async_rdma_port_num": "1", 2024-09-22T03:08:13.531 INFO:teuthology.orchestra.run.smithi153.stdout: "ms_async_rdma_receive_buffers": "32768", 2024-09-22T03:08:13.531 INFO:teuthology.orchestra.run.smithi153.stdout: "ms_async_rdma_receive_queue_len": "4096", 2024-09-22T03:08:13.531 INFO:teuthology.orchestra.run.smithi153.stdout: "ms_async_rdma_roce_ver": "1", 2024-09-22T03:08:13.531 INFO:teuthology.orchestra.run.smithi153.stdout: "ms_async_rdma_send_buffers": "1024", 2024-09-22T03:08:13.531 INFO:teuthology.orchestra.run.smithi153.stdout: "ms_async_rdma_sl": "3", 2024-09-22T03:08:13.531 INFO:teuthology.orchestra.run.smithi153.stdout: "ms_async_rdma_support_srq": "true", 2024-09-22T03:08:13.531 INFO:teuthology.orchestra.run.smithi153.stdout: "ms_async_rdma_type": "ib", 2024-09-22T03:08:13.531 INFO:teuthology.orchestra.run.smithi153.stdout: "ms_async_reap_threshold": "5", 2024-09-22T03:08:13.531 INFO:teuthology.orchestra.run.smithi153.stdout: "ms_bind_before_connect": "false", 2024-09-22T03:08:13.531 INFO:teuthology.orchestra.run.smithi153.stdout: "ms_bind_ipv4": "true", 2024-09-22T03:08:13.531 INFO:teuthology.orchestra.run.smithi153.stdout: "ms_bind_ipv6": "false", 2024-09-22T03:08:13.531 INFO:teuthology.orchestra.run.smithi153.stdout: "ms_bind_msgr1": "true", 2024-09-22T03:08:13.532 INFO:teuthology.orchestra.run.smithi153.stdout: "ms_bind_msgr2": "true", 2024-09-22T03:08:13.532 INFO:teuthology.orchestra.run.smithi153.stdout: "ms_bind_port_max": "7568", 2024-09-22T03:08:13.532 INFO:teuthology.orchestra.run.smithi153.stdout: "ms_bind_port_min": "6800", 2024-09-22T03:08:13.532 INFO:teuthology.orchestra.run.smithi153.stdout: "ms_bind_prefer_ipv4": "false", 2024-09-22T03:08:13.532 INFO:teuthology.orchestra.run.smithi153.stdout: "ms_bind_retry_count": "3", 2024-09-22T03:08:13.532 INFO:teuthology.orchestra.run.smithi153.stdout: "ms_bind_retry_delay": "5", 2024-09-22T03:08:13.532 INFO:teuthology.orchestra.run.smithi153.stdout: "ms_blackhole_client": "false", 2024-09-22T03:08:13.532 INFO:teuthology.orchestra.run.smithi153.stdout: "ms_blackhole_mds": "false", 2024-09-22T03:08:13.532 INFO:teuthology.orchestra.run.smithi153.stdout: "ms_blackhole_mgr": "false", 2024-09-22T03:08:13.532 INFO:teuthology.orchestra.run.smithi153.stdout: "ms_blackhole_mon": "false", 2024-09-22T03:08:13.532 INFO:teuthology.orchestra.run.smithi153.stdout: "ms_blackhole_osd": "false", 2024-09-22T03:08:13.532 INFO:teuthology.orchestra.run.smithi153.stdout: "ms_client_mode": "crc", 2024-09-22T03:08:13.532 INFO:teuthology.orchestra.run.smithi153.stdout: "ms_cluster_mode": "crc", 2024-09-22T03:08:13.532 INFO:teuthology.orchestra.run.smithi153.stdout: "ms_cluster_type": "", 2024-09-22T03:08:13.532 INFO:teuthology.orchestra.run.smithi153.stdout: "ms_compress_secure": "false", 2024-09-22T03:08:13.533 INFO:teuthology.orchestra.run.smithi153.stdout: "ms_connection_idle_timeout": "900", 2024-09-22T03:08:13.533 INFO:teuthology.orchestra.run.smithi153.stdout: "ms_connection_ready_timeout": "10", 2024-09-22T03:08:13.533 INFO:teuthology.orchestra.run.smithi153.stdout: "ms_crc_data": "true", 2024-09-22T03:08:13.533 INFO:teuthology.orchestra.run.smithi153.stdout: "ms_crc_header": "true", 2024-09-22T03:08:13.533 INFO:teuthology.orchestra.run.smithi153.stdout: "ms_die_on_bad_msg": "false", 2024-09-22T03:08:13.533 INFO:teuthology.orchestra.run.smithi153.stdout: "ms_die_on_bug": "true", 2024-09-22T03:08:13.533 INFO:teuthology.orchestra.run.smithi153.stdout: "ms_die_on_old_message": "true", 2024-09-22T03:08:13.533 INFO:teuthology.orchestra.run.smithi153.stdout: "ms_die_on_skipped_message": "false", 2024-09-22T03:08:13.533 INFO:teuthology.orchestra.run.smithi153.stdout: "ms_die_on_unhandled_msg": "false", 2024-09-22T03:08:13.533 INFO:teuthology.orchestra.run.smithi153.stdout: "ms_dispatch_throttle_bytes": "104857600", 2024-09-22T03:08:13.533 INFO:teuthology.orchestra.run.smithi153.stdout: "ms_dpdk_coremask": "0xF", 2024-09-22T03:08:13.533 INFO:teuthology.orchestra.run.smithi153.stdout: "ms_dpdk_debug_allow_loopback": "false", 2024-09-22T03:08:13.533 INFO:teuthology.orchestra.run.smithi153.stdout: "ms_dpdk_devs_allowlist": "", 2024-09-22T03:08:13.533 INFO:teuthology.orchestra.run.smithi153.stdout: "ms_dpdk_enable_tso": "true", 2024-09-22T03:08:13.534 INFO:teuthology.orchestra.run.smithi153.stdout: "ms_dpdk_gateway_ipv4_addr": "", 2024-09-22T03:08:13.534 INFO:teuthology.orchestra.run.smithi153.stdout: "ms_dpdk_host_ipv4_addr": "", 2024-09-22T03:08:13.534 INFO:teuthology.orchestra.run.smithi153.stdout: "ms_dpdk_hugepages": "", 2024-09-22T03:08:13.534 INFO:teuthology.orchestra.run.smithi153.stdout: "ms_dpdk_hw_flow_control": "true", 2024-09-22T03:08:13.534 INFO:teuthology.orchestra.run.smithi153.stdout: "ms_dpdk_hw_queue_weight": "1.000000", 2024-09-22T03:08:13.534 INFO:teuthology.orchestra.run.smithi153.stdout: "ms_dpdk_lro": "true", 2024-09-22T03:08:13.534 INFO:teuthology.orchestra.run.smithi153.stdout: "ms_dpdk_memory_channel": "4", 2024-09-22T03:08:13.534 INFO:teuthology.orchestra.run.smithi153.stdout: "ms_dpdk_netmask_ipv4_addr": "", 2024-09-22T03:08:13.534 INFO:teuthology.orchestra.run.smithi153.stdout: "ms_dpdk_pmd": "", 2024-09-22T03:08:13.534 INFO:teuthology.orchestra.run.smithi153.stdout: "ms_dpdk_port_id": "0", 2024-09-22T03:08:13.534 INFO:teuthology.orchestra.run.smithi153.stdout: "ms_dpdk_rx_buffer_count_per_core": "8192", 2024-09-22T03:08:13.534 INFO:teuthology.orchestra.run.smithi153.stdout: "ms_dump_corrupt_message_level": "1", 2024-09-22T03:08:13.534 INFO:teuthology.orchestra.run.smithi153.stdout: "ms_dump_on_send": "false", 2024-09-22T03:08:13.535 INFO:teuthology.orchestra.run.smithi153.stdout: "ms_initial_backoff": "0.200000", 2024-09-22T03:08:13.535 INFO:teuthology.orchestra.run.smithi153.stdout: "ms_inject_delay_max": "1.000000", 2024-09-22T03:08:13.535 INFO:teuthology.orchestra.run.smithi153.stdout: "ms_inject_delay_probability": "0.000000", 2024-09-22T03:08:13.535 INFO:teuthology.orchestra.run.smithi153.stdout: "ms_inject_delay_type": "", 2024-09-22T03:08:13.535 INFO:teuthology.orchestra.run.smithi153.stdout: "ms_inject_internal_delays": "0.000000", 2024-09-22T03:08:13.535 INFO:teuthology.orchestra.run.smithi153.stdout: "ms_inject_network_congestion": "0", 2024-09-22T03:08:13.535 INFO:teuthology.orchestra.run.smithi153.stdout: "ms_inject_socket_failures": "0", 2024-09-22T03:08:13.535 INFO:teuthology.orchestra.run.smithi153.stdout: "ms_learn_addr_from_peer": "true", 2024-09-22T03:08:13.535 INFO:teuthology.orchestra.run.smithi153.stdout: "ms_max_accept_failures": "4", 2024-09-22T03:08:13.535 INFO:teuthology.orchestra.run.smithi153.stdout: "ms_max_backoff": "15.000000", 2024-09-22T03:08:13.535 INFO:teuthology.orchestra.run.smithi153.stdout: "ms_mon_client_mode": "crc", 2024-09-22T03:08:13.535 INFO:teuthology.orchestra.run.smithi153.stdout: "ms_mon_cluster_mode": "crc", 2024-09-22T03:08:13.535 INFO:teuthology.orchestra.run.smithi153.stdout: "ms_mon_service_mode": "crc", 2024-09-22T03:08:13.536 INFO:teuthology.orchestra.run.smithi153.stdout: "ms_osd_compress_min_size": "1024", 2024-09-22T03:08:13.536 INFO:teuthology.orchestra.run.smithi153.stdout: "ms_osd_compress_mode": "none", 2024-09-22T03:08:13.536 INFO:teuthology.orchestra.run.smithi153.stdout: "ms_osd_compression_algorithm": "snappy", 2024-09-22T03:08:13.536 INFO:teuthology.orchestra.run.smithi153.stdout: "ms_pq_max_tokens_per_priority": "16777216", 2024-09-22T03:08:13.536 INFO:teuthology.orchestra.run.smithi153.stdout: "ms_pq_min_cost": "65536", 2024-09-22T03:08:13.536 INFO:teuthology.orchestra.run.smithi153.stdout: "ms_public_type": "", 2024-09-22T03:08:13.536 INFO:teuthology.orchestra.run.smithi153.stdout: "ms_service_mode": "crc", 2024-09-22T03:08:13.536 INFO:teuthology.orchestra.run.smithi153.stdout: "ms_tcp_listen_backlog": "512", 2024-09-22T03:08:13.536 INFO:teuthology.orchestra.run.smithi153.stdout: "ms_tcp_nodelay": "true", 2024-09-22T03:08:13.536 INFO:teuthology.orchestra.run.smithi153.stdout: "ms_tcp_prefetch_max_size": "4096", 2024-09-22T03:08:13.536 INFO:teuthology.orchestra.run.smithi153.stdout: "ms_tcp_rcvbuf": "0", 2024-09-22T03:08:13.536 INFO:teuthology.orchestra.run.smithi153.stdout: "ms_type": "async+posix", 2024-09-22T03:08:13.536 INFO:teuthology.orchestra.run.smithi153.stdout: "no_config_file": "false", 2024-09-22T03:08:13.537 INFO:teuthology.orchestra.run.smithi153.stdout: "objecter_completion_locks_per_session": "32", 2024-09-22T03:08:13.537 INFO:teuthology.orchestra.run.smithi153.stdout: "objecter_debug_inject_relock_delay": "false", 2024-09-22T03:08:13.537 INFO:teuthology.orchestra.run.smithi153.stdout: "objecter_inflight_op_bytes": "104857600", 2024-09-22T03:08:13.537 INFO:teuthology.orchestra.run.smithi153.stdout: "objecter_inflight_ops": "1024", 2024-09-22T03:08:13.537 INFO:teuthology.orchestra.run.smithi153.stdout: "objecter_inject_no_watch_ping": "false", 2024-09-22T03:08:13.537 INFO:teuthology.orchestra.run.smithi153.stdout: "objecter_retry_writes_after_first_reply": "false", 2024-09-22T03:08:13.537 INFO:teuthology.orchestra.run.smithi153.stdout: "objecter_tick_interval": "5.000000", 2024-09-22T03:08:13.537 INFO:teuthology.orchestra.run.smithi153.stdout: "objecter_timeout": "10.000000", 2024-09-22T03:08:13.537 INFO:teuthology.orchestra.run.smithi153.stdout: "objectstore_blackhole": "false", 2024-09-22T03:08:13.537 INFO:teuthology.orchestra.run.smithi153.stdout: "objectstore_debug_throw_on_failed_txc": "false", 2024-09-22T03:08:13.537 INFO:teuthology.orchestra.run.smithi153.stdout: "openssl_engine_opts": "", 2024-09-22T03:08:13.537 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_agent_delay_time": "5.000000", 2024-09-22T03:08:13.537 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_agent_hist_halflife": "1000", 2024-09-22T03:08:13.538 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_agent_max_low_ops": "2", 2024-09-22T03:08:13.538 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_agent_max_ops": "4", 2024-09-22T03:08:13.538 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_agent_min_evict_effort": "0.100000", 2024-09-22T03:08:13.538 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_agent_quantize_effort": "0.100000", 2024-09-22T03:08:13.538 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_agent_slop": "0.020000", 2024-09-22T03:08:13.538 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_aggregated_slow_ops_logging": "true", 2024-09-22T03:08:13.538 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_allow_recovery_below_min_size": "true", 2024-09-22T03:08:13.538 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_asio_thread_count": "2", 2024-09-22T03:08:13.538 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_async_recovery_min_cost": "100", 2024-09-22T03:08:13.538 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_auto_mark_unfound_lost": "false", 2024-09-22T03:08:13.538 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_backfill_retry_interval": "30.000000", 2024-09-22T03:08:13.538 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_backfill_scan_max": "512", 2024-09-22T03:08:13.538 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_backfill_scan_min": "64", 2024-09-22T03:08:13.538 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_backoff_on_degraded": "false", 2024-09-22T03:08:13.539 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_backoff_on_peering": "false", 2024-09-22T03:08:13.539 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_backoff_on_unfound": "true", 2024-09-22T03:08:13.539 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_beacon_report_interval": "300", 2024-09-22T03:08:13.539 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_bench_duration": "30", 2024-09-22T03:08:13.539 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_bench_large_size_max_throughput": "104857600", 2024-09-22T03:08:13.539 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_bench_max_block_size": "67108864", 2024-09-22T03:08:13.539 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_bench_small_size_max_iops": "100", 2024-09-22T03:08:13.539 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_blkin_trace_all": "false", 2024-09-22T03:08:13.539 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_blocked_scrub_grace_period": "120", 2024-09-22T03:08:13.539 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_calc_pg_upmaps_aggressively": "true", 2024-09-22T03:08:13.539 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_calc_pg_upmaps_aggressively_fast": "true", 2024-09-22T03:08:13.539 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_calc_pg_upmaps_local_fallback_retries": "100", 2024-09-22T03:08:13.539 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_check_for_log_corruption": "false", 2024-09-22T03:08:13.540 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_check_max_object_name_len_on_startup": "true", 2024-09-22T03:08:13.540 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_class_default_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-09-22T03:08:13.540 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_class_dir": "/usr/lib64/rados-classes", 2024-09-22T03:08:13.540 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_class_load_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-09-22T03:08:13.540 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_class_update_on_start": "true", 2024-09-22T03:08:13.540 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_client_message_cap": "256", 2024-09-22T03:08:13.540 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_client_message_size_cap": "524288000", 2024-09-22T03:08:13.540 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_client_op_priority": "63", 2024-09-22T03:08:13.540 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_client_watch_timeout": "30", 2024-09-22T03:08:13.540 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_command_max_records": "256", 2024-09-22T03:08:13.540 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_compact_on_start": "false", 2024-09-22T03:08:13.540 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_copyfrom_max_chunk": "8388608", 2024-09-22T03:08:13.540 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_crush_chooseleaf_type": "0", 2024-09-22T03:08:13.541 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_crush_initial_weight": "-1.000000", 2024-09-22T03:08:13.541 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_crush_update_on_start": "true", 2024-09-22T03:08:13.541 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_crush_update_weight_set": "true", 2024-09-22T03:08:13.541 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_data": "/var/lib/ceph/osd/ceph-a", 2024-09-22T03:08:13.541 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_debug_crash_on_ignored_backoff": "false", 2024-09-22T03:08:13.541 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_debug_deep_scrub_sleep": "0.000000", 2024-09-22T03:08:13.541 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_debug_drop_ping_duration": "0", 2024-09-22T03:08:13.541 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_debug_drop_ping_probability": "0.000000", 2024-09-22T03:08:13.541 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_debug_feed_pullee": "-1", 2024-09-22T03:08:13.541 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_debug_inject_copyfrom_error": "false", 2024-09-22T03:08:13.541 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_debug_inject_dispatch_delay_duration": "0.100000", 2024-09-22T03:08:13.541 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_debug_inject_dispatch_delay_probability": "0.000000", 2024-09-22T03:08:13.541 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_debug_misdirected_ops": "false", 2024-09-22T03:08:13.542 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_debug_no_acting_change": "false", 2024-09-22T03:08:13.542 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_debug_no_purge_strays": "false", 2024-09-22T03:08:13.542 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_debug_op_order": "false", 2024-09-22T03:08:13.542 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_debug_pg_log_writeout": "false", 2024-09-22T03:08:13.542 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_debug_pretend_recovery_active": "false", 2024-09-22T03:08:13.542 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_debug_random_push_read_error": "0.000000", 2024-09-22T03:08:13.542 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_debug_reject_backfill_probability": "0.000000", 2024-09-22T03:08:13.542 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_debug_shutdown": "false", 2024-09-22T03:08:13.542 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_debug_skip_full_check_in_backfill_reservation": "false", 2024-09-22T03:08:13.542 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_debug_skip_full_check_in_recovery": "false", 2024-09-22T03:08:13.542 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_debug_trim_objects": "false", 2024-09-22T03:08:13.542 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_debug_verify_cached_snaps": "false", 2024-09-22T03:08:13.542 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_debug_verify_missing_on_start": "false", 2024-09-22T03:08:13.543 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_debug_verify_snaps": "false", 2024-09-22T03:08:13.543 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_debug_verify_stray_on_activate": "false", 2024-09-22T03:08:13.543 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_deep_scrub_interval": "604800.000000", 2024-09-22T03:08:13.543 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_deep_scrub_keys": "1024", 2024-09-22T03:08:13.543 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_deep_scrub_large_omap_object_key_threshold": "200000", 2024-09-22T03:08:13.543 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_deep_scrub_large_omap_object_value_sum_threshold": "1073741824", 2024-09-22T03:08:13.543 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_deep_scrub_randomize_ratio": "0.150000", 2024-09-22T03:08:13.543 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_deep_scrub_stride": "524288", 2024-09-22T03:08:13.543 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_deep_scrub_update_digest_min_age": "7200", 2024-09-22T03:08:13.543 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_default_data_pool_replay_window": "5", 2024-09-22T03:08:13.543 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_default_notify_timeout": "30", 2024-09-22T03:08:13.543 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_delete_sleep": "0.000000", 2024-09-22T03:08:13.543 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_delete_sleep_hdd": "5.000000", 2024-09-22T03:08:13.544 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_delete_sleep_hybrid": "1.000000", 2024-09-22T03:08:13.544 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_delete_sleep_ssd": "1.000000", 2024-09-22T03:08:13.544 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_discard_disconnected_ops": "true", 2024-09-22T03:08:13.544 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_enable_op_tracker": "true", 2024-09-22T03:08:13.544 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_erasure_code_plugins": "jerasure lrc", 2024-09-22T03:08:13.544 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_extblkdev_plugins": "vdo", 2024-09-22T03:08:13.544 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_failsafe_full_ratio": "0.970000", 2024-09-22T03:08:13.544 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_fast_fail_on_connection_refused": "true", 2024-09-22T03:08:13.544 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_fast_info": "true", 2024-09-22T03:08:13.544 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_fast_shutdown": "true", 2024-09-22T03:08:13.544 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_fast_shutdown_notify_mon": "true", 2024-09-22T03:08:13.544 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_fast_shutdown_timeout": "15", 2024-09-22T03:08:13.544 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_find_best_info_ignore_history_les": "false", 2024-09-22T03:08:13.545 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_force_auth_primary_missing_objects": "100", 2024-09-22T03:08:13.545 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_force_recovery_pg_log_entries_factor": "1.300000", 2024-09-22T03:08:13.545 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_function_tracing": "false", 2024-09-22T03:08:13.545 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_heartbeat_grace": "20", 2024-09-22T03:08:13.545 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_heartbeat_interval": "6", 2024-09-22T03:08:13.545 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_heartbeat_min_healthy_ratio": "0.330000", 2024-09-22T03:08:13.545 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_heartbeat_min_peers": "10", 2024-09-22T03:08:13.545 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_heartbeat_min_size": "2000", 2024-09-22T03:08:13.545 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_heartbeat_stale": "600", 2024-09-22T03:08:13.545 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_heartbeat_use_min_delay_socket": "false", 2024-09-22T03:08:13.545 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_hit_set_max_size": "100000", 2024-09-22T03:08:13.545 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_hit_set_min_size": "1000", 2024-09-22T03:08:13.545 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_hit_set_namespace": ".ceph-internal", 2024-09-22T03:08:13.545 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_ignore_stale_divergent_priors": "false", 2024-09-22T03:08:13.546 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_inject_bad_map_crc_probability": "0.000000", 2024-09-22T03:08:13.546 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_inject_failure_on_pg_removal": "false", 2024-09-22T03:08:13.546 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_journal": "/var/lib/ceph/osd/ceph-a/journal", 2024-09-22T03:08:13.546 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_journal_flush_on_shutdown": "true", 2024-09-22T03:08:13.546 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_journal_size": "5120", 2024-09-22T03:08:13.546 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_kill_backfill_at": "0", 2024-09-22T03:08:13.546 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_loop_before_reset_tphandle": "64", 2024-09-22T03:08:13.546 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_map_cache_size": "50", 2024-09-22T03:08:13.546 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_map_dedup": "true", 2024-09-22T03:08:13.546 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_map_message_max": "40", 2024-09-22T03:08:13.546 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_map_message_max_bytes": "10485760", 2024-09-22T03:08:13.546 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_map_share_max_epochs": "40", 2024-09-22T03:08:13.546 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_max_attr_name_len": "100", 2024-09-22T03:08:13.547 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_max_attr_size": "0", 2024-09-22T03:08:13.547 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_max_backfills": "1", 2024-09-22T03:08:13.547 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_max_markdown_count": "5", 2024-09-22T03:08:13.547 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_max_markdown_period": "600", 2024-09-22T03:08:13.547 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_max_object_name_len": "2048", 2024-09-22T03:08:13.547 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_max_object_namespace_len": "256", 2024-09-22T03:08:13.547 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_max_object_size": "134217728", 2024-09-22T03:08:13.547 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_max_omap_bytes_per_request": "1073741824", 2024-09-22T03:08:13.547 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_max_omap_entries_per_request": "1024", 2024-09-22T03:08:13.547 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_max_pg_blocked_by": "16", 2024-09-22T03:08:13.547 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_max_pg_log_entries": "10000", 2024-09-22T03:08:13.547 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_max_pg_per_osd_hard_ratio": "3.000000", 2024-09-22T03:08:13.547 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_max_pgls": "1024", 2024-09-22T03:08:13.548 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_max_push_cost": "8388608", 2024-09-22T03:08:13.548 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_max_push_objects": "10", 2024-09-22T03:08:13.548 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_max_scrubs": "3", 2024-09-22T03:08:13.548 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_max_snap_prune_intervals_per_epoch": "512", 2024-09-22T03:08:13.548 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_max_trimming_pgs": "2", 2024-09-22T03:08:13.548 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_max_write_op_reply_len": "64", 2024-09-22T03:08:13.548 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_max_write_size": "90", 2024-09-22T03:08:13.548 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_mclock_force_run_benchmark_on_init": "false", 2024-09-22T03:08:13.548 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_mclock_iops_capacity_threshold_hdd": "500.000000", 2024-09-22T03:08:13.548 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_mclock_iops_capacity_threshold_ssd": "80000.000000", 2024-09-22T03:08:13.548 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_mclock_max_capacity_iops_hdd": "315.000000", 2024-09-22T03:08:13.548 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_mclock_max_capacity_iops_ssd": "21500.000000", 2024-09-22T03:08:13.549 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_mclock_max_sequential_bandwidth_hdd": "157286400", 2024-09-22T03:08:13.549 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_mclock_max_sequential_bandwidth_ssd": "1258291200", 2024-09-22T03:08:13.549 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_mclock_override_recovery_settings": "false", 2024-09-22T03:08:13.549 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_mclock_profile": "balanced", 2024-09-22T03:08:13.549 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_mclock_scheduler_anticipation_timeout": "0.000000", 2024-09-22T03:08:13.549 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_mclock_scheduler_background_best_effort_lim": "0.000000", 2024-09-22T03:08:13.549 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_mclock_scheduler_background_best_effort_res": "0.000000", 2024-09-22T03:08:13.549 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_mclock_scheduler_background_best_effort_wgt": "1", 2024-09-22T03:08:13.549 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_mclock_scheduler_background_recovery_lim": "0.000000", 2024-09-22T03:08:13.549 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_mclock_scheduler_background_recovery_res": "0.000000", 2024-09-22T03:08:13.549 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_mclock_scheduler_background_recovery_wgt": "1", 2024-09-22T03:08:13.549 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_mclock_scheduler_client_lim": "0.000000", 2024-09-22T03:08:13.549 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_mclock_scheduler_client_res": "0.000000", 2024-09-22T03:08:13.549 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_mclock_scheduler_client_wgt": "1", 2024-09-22T03:08:13.550 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_mclock_skip_benchmark": "false", 2024-09-22T03:08:13.550 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_memory_base": "805306368", 2024-09-22T03:08:13.550 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_memory_cache_min": "134217728", 2024-09-22T03:08:13.550 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_memory_cache_resize_interval": "1.000000", 2024-09-22T03:08:13.550 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_memory_expected_fragmentation": "0.150000", 2024-09-22T03:08:13.550 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_memory_target": "4294967296", 2024-09-22T03:08:13.550 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_memory_target_autotune": "false", 2024-09-22T03:08:13.550 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_memory_target_cgroup_limit_ratio": "0.800000", 2024-09-22T03:08:13.550 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_min_pg_log_entries": "250", 2024-09-22T03:08:13.550 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_min_recovery_priority": "0", 2024-09-22T03:08:13.550 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_mon_heartbeat_interval": "30", 2024-09-22T03:08:13.551 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_mon_heartbeat_stat_stale": "3600", 2024-09-22T03:08:13.551 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_mon_report_interval": "5", 2024-09-22T03:08:13.551 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_mon_report_max_in_flight": "2", 2024-09-22T03:08:13.551 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_mon_shutdown_timeout": "5.000000", 2024-09-22T03:08:13.551 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_num_cache_shards": "32", 2024-09-22T03:08:13.551 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_num_op_tracker_shard": "32", 2024-09-22T03:08:13.551 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_numa_auto_affinity": "true", 2024-09-22T03:08:13.551 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_numa_node": "-1", 2024-09-22T03:08:13.551 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_numa_prefer_iface": "true", 2024-09-22T03:08:13.551 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_object_clean_region_max_num_intervals": "10", 2024-09-22T03:08:13.551 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_objecter_finishers": "1", 2024-09-22T03:08:13.551 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_objectstore": "bluestore", 2024-09-22T03:08:13.551 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_objectstore_fuse": "false", 2024-09-22T03:08:13.552 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_objectstore_tracing": "false", 2024-09-22T03:08:13.552 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_op_complaint_time": "30.000000", 2024-09-22T03:08:13.552 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_op_history_duration": "600", 2024-09-22T03:08:13.552 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_op_history_size": "20", 2024-09-22T03:08:13.552 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_op_history_slow_op_size": "20", 2024-09-22T03:08:13.552 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_op_history_slow_op_threshold": "10.000000", 2024-09-22T03:08:13.552 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_op_log_threshold": "5", 2024-09-22T03:08:13.552 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_op_num_shards": "0", 2024-09-22T03:08:13.552 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_op_num_shards_hdd": "5", 2024-09-22T03:08:13.552 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_op_num_shards_ssd": "8", 2024-09-22T03:08:13.552 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_op_num_threads_per_shard": "0", 2024-09-22T03:08:13.552 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_op_num_threads_per_shard_hdd": "1", 2024-09-22T03:08:13.552 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_op_num_threads_per_shard_ssd": "2", 2024-09-22T03:08:13.552 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_op_pq_max_tokens_per_priority": "4194304", 2024-09-22T03:08:13.553 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_op_pq_min_cost": "65536", 2024-09-22T03:08:13.553 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_op_queue": "mclock_scheduler", 2024-09-22T03:08:13.553 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_op_queue_cut_off": "high", 2024-09-22T03:08:13.553 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_op_thread_suicide_timeout": "150", 2024-09-22T03:08:13.553 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_op_thread_timeout": "15", 2024-09-22T03:08:13.553 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_open_classes_on_start": "true", 2024-09-22T03:08:13.553 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_os_flags": "0", 2024-09-22T03:08:13.553 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_peering_op_priority": "255", 2024-09-22T03:08:13.553 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_pg_delete_cost": "1048576", 2024-09-22T03:08:13.553 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_pg_delete_priority": "5", 2024-09-22T03:08:13.553 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_pg_epoch_max_lag_factor": "2.000000", 2024-09-22T03:08:13.553 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_pg_epoch_persisted_max_stale": "40", 2024-09-22T03:08:13.553 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_pg_log_dups_tracked": "3000", 2024-09-22T03:08:13.554 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_pg_log_trim_max": "10000", 2024-09-22T03:08:13.554 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_pg_log_trim_min": "100", 2024-09-22T03:08:13.554 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_pg_max_concurrent_snap_trims": "2", 2024-09-22T03:08:13.554 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_pg_object_context_cache_count": "64", 2024-09-22T03:08:13.554 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_pg_stat_report_interval_max_epochs": "500", 2024-09-22T03:08:13.554 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_pg_stat_report_interval_max_seconds": "5", 2024-09-22T03:08:13.554 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_pool_default_cache_max_evict_check_size": "10", 2024-09-22T03:08:13.554 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_pool_default_cache_min_evict_age": "0", 2024-09-22T03:08:13.554 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_pool_default_cache_min_flush_age": "0", 2024-09-22T03:08:13.554 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_pool_default_cache_target_dirty_high_ratio": "0.600000", 2024-09-22T03:08:13.554 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_pool_default_cache_target_dirty_ratio": "0.400000", 2024-09-22T03:08:13.554 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_pool_default_cache_target_full_ratio": "0.800000", 2024-09-22T03:08:13.554 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_pool_default_crimson": "true", 2024-09-22T03:08:13.554 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_pool_default_crush_rule": "-1", 2024-09-22T03:08:13.555 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_pool_default_ec_fast_read": "false", 2024-09-22T03:08:13.555 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_pool_default_erasure_code_profile": "plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd", 2024-09-22T03:08:13.555 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_pool_default_flag_bulk": "false", 2024-09-22T03:08:13.555 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_pool_default_flag_hashpspool": "true", 2024-09-22T03:08:13.555 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_pool_default_flag_nodelete": "false", 2024-09-22T03:08:13.555 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_pool_default_flag_nopgchange": "false", 2024-09-22T03:08:13.555 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_pool_default_flag_nosizechange": "false", 2024-09-22T03:08:13.555 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_pool_default_flags": "0", 2024-09-22T03:08:13.555 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_pool_default_hit_set_bloom_fpp": "0.050000", 2024-09-22T03:08:13.555 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_pool_default_min_size": "2", 2024-09-22T03:08:13.555 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_pool_default_pg_autoscale_mode": "off", 2024-09-22T03:08:13.555 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_pool_default_pg_num": "32", 2024-09-22T03:08:13.556 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_pool_default_pgp_num": "0", 2024-09-22T03:08:13.556 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_pool_default_read_lease_ratio": "0.800000", 2024-09-22T03:08:13.556 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_pool_default_read_ratio": "70", 2024-09-22T03:08:13.556 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_pool_default_size": "3", 2024-09-22T03:08:13.556 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_pool_default_type": "replicated", 2024-09-22T03:08:13.556 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_pool_erasure_code_stripe_unit": "4096", 2024-09-22T03:08:13.556 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_pool_use_gmt_hitset": "true", 2024-09-22T03:08:13.556 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_push_per_object_cost": "1000", 2024-09-22T03:08:13.556 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_read_ec_check_for_errors": "false", 2024-09-22T03:08:13.556 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_recover_clone_overlap": "true", 2024-09-22T03:08:13.556 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_recover_clone_overlap_limit": "10", 2024-09-22T03:08:13.556 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_recovery_cost": "20971520", 2024-09-22T03:08:13.556 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_recovery_delay_start": "0.000000", 2024-09-22T03:08:13.557 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_recovery_max_active": "0", 2024-09-22T03:08:13.557 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_recovery_max_active_hdd": "3", 2024-09-22T03:08:13.557 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_recovery_max_active_ssd": "10", 2024-09-22T03:08:13.557 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_recovery_max_chunk": "8388608", 2024-09-22T03:08:13.557 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_recovery_max_omap_entries_per_chunk": "8096", 2024-09-22T03:08:13.557 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_recovery_max_single_start": "1", 2024-09-22T03:08:13.557 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_recovery_op_priority": "3", 2024-09-22T03:08:13.557 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_recovery_op_warn_multiple": "16", 2024-09-22T03:08:13.557 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_recovery_priority": "5", 2024-09-22T03:08:13.557 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_recovery_retry_interval": "30.000000", 2024-09-22T03:08:13.557 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_recovery_sleep": "0.000000", 2024-09-22T03:08:13.557 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_recovery_sleep_hdd": "0.100000", 2024-09-22T03:08:13.558 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_recovery_sleep_hybrid": "0.025000", 2024-09-22T03:08:13.558 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_recovery_sleep_ssd": "0.000000", 2024-09-22T03:08:13.558 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_repair_during_recovery": "false", 2024-09-22T03:08:13.558 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_requested_scrub_priority": "120", 2024-09-22T03:08:13.558 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_rocksdb_iterator_bounds_enabled": "true", 2024-09-22T03:08:13.558 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_rollback_to_cluster_snap": "", 2024-09-22T03:08:13.558 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_scrub_auto_repair": "false", 2024-09-22T03:08:13.558 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_scrub_auto_repair_num_errors": "5", 2024-09-22T03:08:13.558 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_scrub_backoff_ratio": "0.660000", 2024-09-22T03:08:13.558 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_scrub_begin_hour": "0", 2024-09-22T03:08:13.558 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_scrub_begin_week_day": "0", 2024-09-22T03:08:13.558 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_scrub_chunk_max": "25", 2024-09-22T03:08:13.559 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_scrub_chunk_min": "5", 2024-09-22T03:08:13.559 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_scrub_cost": "52428800", 2024-09-22T03:08:13.559 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_scrub_disable_reservation_queuing": "false", 2024-09-22T03:08:13.559 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_scrub_during_recovery": "false", 2024-09-22T03:08:13.559 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_scrub_end_hour": "0", 2024-09-22T03:08:13.559 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_scrub_end_week_day": "0", 2024-09-22T03:08:13.559 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_scrub_event_cost": "4096", 2024-09-22T03:08:13.559 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_scrub_extended_sleep": "0.000000", 2024-09-22T03:08:13.559 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_scrub_interval_randomize_ratio": "0.500000", 2024-09-22T03:08:13.559 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_scrub_invalid_stats": "true", 2024-09-22T03:08:13.559 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_scrub_load_threshold": "0.500000", 2024-09-22T03:08:13.559 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_scrub_max_interval": "604800.000000", 2024-09-22T03:08:13.559 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_scrub_max_preemptions": "5", 2024-09-22T03:08:13.560 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_scrub_min_interval": "86400.000000", 2024-09-22T03:08:13.560 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_scrub_priority": "5", 2024-09-22T03:08:13.560 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_scrub_sleep": "0.000000", 2024-09-22T03:08:13.560 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_shallow_scrub_chunk_max": "100", 2024-09-22T03:08:13.560 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_shallow_scrub_chunk_min": "50", 2024-09-22T03:08:13.560 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_shutdown_pgref_assert": "false", 2024-09-22T03:08:13.560 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_skip_data_digest": "false", 2024-09-22T03:08:13.560 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_smart_report_timeout": "5", 2024-09-22T03:08:13.560 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_snap_trim_cost": "1048576", 2024-09-22T03:08:13.560 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_snap_trim_priority": "5", 2024-09-22T03:08:13.560 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_snap_trim_sleep": "0.000000", 2024-09-22T03:08:13.560 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_snap_trim_sleep_hdd": "5.000000", 2024-09-22T03:08:13.560 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_snap_trim_sleep_hybrid": "2.000000", 2024-09-22T03:08:13.560 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_snap_trim_sleep_ssd": "0.000000", 2024-09-22T03:08:13.561 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_stats_update_period_not_scrubbing": "120", 2024-09-22T03:08:13.561 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_stats_update_period_scrubbing": "15", 2024-09-22T03:08:13.561 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_target_pg_log_entries_per_osd": "300000", 2024-09-22T03:08:13.561 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_target_transaction_size": "30", 2024-09-22T03:08:13.561 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_tier_default_cache_hit_set_count": "4", 2024-09-22T03:08:13.561 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_tier_default_cache_hit_set_grade_decay_rate": "20", 2024-09-22T03:08:13.561 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_tier_default_cache_hit_set_period": "1200", 2024-09-22T03:08:13.561 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_tier_default_cache_hit_set_search_last_n": "1", 2024-09-22T03:08:13.561 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_tier_default_cache_hit_set_type": "bloom", 2024-09-22T03:08:13.561 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_tier_default_cache_min_read_recency_for_promote": "1", 2024-09-22T03:08:13.561 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_tier_default_cache_min_write_recency_for_promote": "1", 2024-09-22T03:08:13.561 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_tier_default_cache_mode": "writeback", 2024-09-22T03:08:13.561 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_tier_promote_max_bytes_sec": "5242880", 2024-09-22T03:08:13.562 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_tier_promote_max_objects_sec": "25", 2024-09-22T03:08:13.562 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_tracing": "false", 2024-09-22T03:08:13.562 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_use_stale_snap": "false", 2024-09-22T03:08:13.562 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_uuid": "00000000-0000-0000-0000-000000000000", 2024-09-22T03:08:13.562 INFO:teuthology.orchestra.run.smithi153.stdout: "osdc_blkin_trace_all": "false", 2024-09-22T03:08:13.562 INFO:teuthology.orchestra.run.smithi153.stdout: "paxos_kill_at": "0", 2024-09-22T03:08:13.562 INFO:teuthology.orchestra.run.smithi153.stdout: "paxos_max_join_drift": "10", 2024-09-22T03:08:13.562 INFO:teuthology.orchestra.run.smithi153.stdout: "paxos_min": "500", 2024-09-22T03:08:13.562 INFO:teuthology.orchestra.run.smithi153.stdout: "paxos_min_wait": "0.050000", 2024-09-22T03:08:13.562 INFO:teuthology.orchestra.run.smithi153.stdout: "paxos_propose_interval": "1.000000", 2024-09-22T03:08:13.562 INFO:teuthology.orchestra.run.smithi153.stdout: "paxos_service_trim_max": "500", 2024-09-22T03:08:13.562 INFO:teuthology.orchestra.run.smithi153.stdout: "paxos_service_trim_max_multiplier": "20", 2024-09-22T03:08:13.563 INFO:teuthology.orchestra.run.smithi153.stdout: "paxos_service_trim_min": "10", 2024-09-22T03:08:13.563 INFO:teuthology.orchestra.run.smithi153.stdout: "paxos_stash_full_interval": "25", 2024-09-22T03:08:13.563 INFO:teuthology.orchestra.run.smithi153.stdout: "paxos_trim_max": "500", 2024-09-22T03:08:13.563 INFO:teuthology.orchestra.run.smithi153.stdout: "paxos_trim_min": "250", 2024-09-22T03:08:13.563 INFO:teuthology.orchestra.run.smithi153.stdout: "perf": "true", 2024-09-22T03:08:13.563 INFO:teuthology.orchestra.run.smithi153.stdout: "pid_file": "/var/run/ceph/ceph-mon.a.pid", 2024-09-22T03:08:13.563 INFO:teuthology.orchestra.run.smithi153.stdout: "plugin_crypto_accelerator": "crypto_isal", 2024-09-22T03:08:13.563 INFO:teuthology.orchestra.run.smithi153.stdout: "plugin_dir": "/usr/lib64/ceph", 2024-09-22T03:08:13.563 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "-", 2024-09-22T03:08:13.563 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrv": "", 2024-09-22T03:08:13.563 INFO:teuthology.orchestra.run.smithi153.stdout: "public_bind_addr": "-", 2024-09-22T03:08:13.563 INFO:teuthology.orchestra.run.smithi153.stdout: "public_network": "", 2024-09-22T03:08:13.563 INFO:teuthology.orchestra.run.smithi153.stdout: "public_network_interface": "", 2024-09-22T03:08:13.564 INFO:teuthology.orchestra.run.smithi153.stdout: "qat_compressor_busy_polling": "false", 2024-09-22T03:08:13.564 INFO:teuthology.orchestra.run.smithi153.stdout: "qat_compressor_enabled": "false", 2024-09-22T03:08:13.564 INFO:teuthology.orchestra.run.smithi153.stdout: "qat_compressor_session_max_number": "256", 2024-09-22T03:08:13.564 INFO:teuthology.orchestra.run.smithi153.stdout: "rados_mon_op_timeout": "0", 2024-09-22T03:08:13.564 INFO:teuthology.orchestra.run.smithi153.stdout: "rados_osd_op_timeout": "0", 2024-09-22T03:08:13.564 INFO:teuthology.orchestra.run.smithi153.stdout: "rados_tracing": "false", 2024-09-22T03:08:13.564 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_atime_update_interval": "60", 2024-09-22T03:08:13.564 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_auto_exclusive_lock_until_manual_request": "true", 2024-09-22T03:08:13.564 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_balance_parent_reads": "false", 2024-09-22T03:08:13.564 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_balance_snap_reads": "false", 2024-09-22T03:08:13.564 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_blkin_trace_all": "false", 2024-09-22T03:08:13.564 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_blocklist_expire_seconds": "0", 2024-09-22T03:08:13.564 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_blocklist_on_break_lock": "true", 2024-09-22T03:08:13.565 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_cache": "true", 2024-09-22T03:08:13.565 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_cache_block_writes_upfront": "false", 2024-09-22T03:08:13.565 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_cache_max_dirty": "25165824", 2024-09-22T03:08:13.565 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_cache_max_dirty_age": "1.000000", 2024-09-22T03:08:13.565 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_cache_max_dirty_object": "0", 2024-09-22T03:08:13.565 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_cache_policy": "writearound", 2024-09-22T03:08:13.565 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_cache_size": "33554432", 2024-09-22T03:08:13.565 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_cache_target_dirty": "16777216", 2024-09-22T03:08:13.565 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_cache_writethrough_until_flush": "true", 2024-09-22T03:08:13.565 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_clone_copy_on_read": "false", 2024-09-22T03:08:13.565 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_compression_hint": "none", 2024-09-22T03:08:13.565 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_concurrent_management_ops": "10", 2024-09-22T03:08:13.565 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_config_pool_override_update_timestamp": "0", 2024-09-22T03:08:13.566 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_default_clone_format": "auto", 2024-09-22T03:08:13.566 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_default_data_pool": "", 2024-09-22T03:08:13.566 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_default_features": "61", 2024-09-22T03:08:13.566 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_default_format": "2", 2024-09-22T03:08:13.566 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_default_map_options": "", 2024-09-22T03:08:13.566 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_default_order": "22", 2024-09-22T03:08:13.566 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_default_pool": "rbd", 2024-09-22T03:08:13.566 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_default_snapshot_quiesce_mode": "required", 2024-09-22T03:08:13.566 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_default_stripe_count": "0", 2024-09-22T03:08:13.566 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_default_stripe_unit": "0", 2024-09-22T03:08:13.566 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_disable_zero_copy_writes": "true", 2024-09-22T03:08:13.566 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_discard_granularity_bytes": "65536", 2024-09-22T03:08:13.566 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_discard_on_zeroed_write_same": "true", 2024-09-22T03:08:13.567 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_enable_alloc_hint": "true", 2024-09-22T03:08:13.567 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_invalidate_object_map_on_timeout": "true", 2024-09-22T03:08:13.567 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_io_scheduler": "simple", 2024-09-22T03:08:13.567 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_io_scheduler_simple_max_delay": "0", 2024-09-22T03:08:13.567 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_journal_commit_age": "5.000000", 2024-09-22T03:08:13.567 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_journal_max_concurrent_object_sets": "0", 2024-09-22T03:08:13.567 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_journal_max_payload_bytes": "16384", 2024-09-22T03:08:13.567 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_journal_object_flush_age": "0.000000", 2024-09-22T03:08:13.567 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_journal_object_flush_bytes": "1048576", 2024-09-22T03:08:13.567 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_journal_object_flush_interval": "0", 2024-09-22T03:08:13.567 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_journal_object_max_in_flight_appends": "0", 2024-09-22T03:08:13.567 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_journal_object_writethrough_until_flush": "true", 2024-09-22T03:08:13.568 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_journal_order": "24", 2024-09-22T03:08:13.568 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_journal_pool": "", 2024-09-22T03:08:13.568 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_journal_splay_width": "4", 2024-09-22T03:08:13.568 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_localize_parent_reads": "false", 2024-09-22T03:08:13.568 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_localize_snap_reads": "false", 2024-09-22T03:08:13.568 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_mirror_concurrent_image_deletions": "1", 2024-09-22T03:08:13.568 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_mirror_concurrent_image_syncs": "5", 2024-09-22T03:08:13.568 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_mirror_delete_retry_interval": "30.000000", 2024-09-22T03:08:13.568 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_mirror_image_perf_stats_prio": "5", 2024-09-22T03:08:13.568 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_mirror_image_policy_migration_throttle": "300", 2024-09-22T03:08:13.568 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_mirror_image_policy_rebalance_timeout": "0.000000", 2024-09-22T03:08:13.568 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_mirror_image_policy_type": "simple", 2024-09-22T03:08:13.568 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_mirror_image_policy_update_throttle_interval": "1.000000", 2024-09-22T03:08:13.569 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_mirror_image_state_check_interval": "30", 2024-09-22T03:08:13.569 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_mirror_journal_commit_age": "5.000000", 2024-09-22T03:08:13.569 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_mirror_journal_poll_age": "5.000000", 2024-09-22T03:08:13.569 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_mirror_leader_heartbeat_interval": "5", 2024-09-22T03:08:13.569 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_mirror_leader_max_acquire_attempts_before_break": "3", 2024-09-22T03:08:13.569 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_mirror_leader_max_missed_heartbeats": "2", 2024-09-22T03:08:13.569 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_mirror_memory_autotune": "true", 2024-09-22T03:08:13.569 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_mirror_memory_base": "805306368", 2024-09-22T03:08:13.569 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_mirror_memory_cache_autotune_interval": "30.000000", 2024-09-22T03:08:13.569 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_mirror_memory_cache_min": "134217728", 2024-09-22T03:08:13.569 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_mirror_memory_cache_resize_interval": "5.000000", 2024-09-22T03:08:13.569 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_mirror_memory_expected_fragmentation": "0.150000", 2024-09-22T03:08:13.569 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_mirror_memory_target": "4294967296", 2024-09-22T03:08:13.570 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_mirror_perf_stats_prio": "5", 2024-09-22T03:08:13.570 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_mirror_pool_replayers_refresh_interval": "30", 2024-09-22T03:08:13.570 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_mirror_sync_point_update_age": "30.000000", 2024-09-22T03:08:13.570 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_mirroring_delete_delay": "0", 2024-09-22T03:08:13.570 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_mirroring_max_mirroring_snapshots": "5", 2024-09-22T03:08:13.570 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_mirroring_replay_delay": "0", 2024-09-22T03:08:13.570 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_mirroring_resync_after_disconnect": "false", 2024-09-22T03:08:13.570 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_move_parent_to_trash_on_remove": "false", 2024-09-22T03:08:13.570 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_move_to_trash_on_remove": "false", 2024-09-22T03:08:13.570 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_move_to_trash_on_remove_expire_seconds": "0", 2024-09-22T03:08:13.570 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_mtime_update_interval": "60", 2024-09-22T03:08:13.570 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_non_blocking_aio": "true", 2024-09-22T03:08:13.571 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_op_thread_timeout": "60", 2024-09-22T03:08:13.571 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_op_threads": "1", 2024-09-22T03:08:13.571 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_parent_cache_enabled": "false", 2024-09-22T03:08:13.571 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_persistent_cache_mode": "disabled", 2024-09-22T03:08:13.571 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_persistent_cache_path": "/tmp", 2024-09-22T03:08:13.571 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_persistent_cache_size": "1073741824", 2024-09-22T03:08:13.571 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_plugins": "", 2024-09-22T03:08:13.571 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_qos_bps_burst": "0", 2024-09-22T03:08:13.571 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_qos_bps_burst_seconds": "1", 2024-09-22T03:08:13.571 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_qos_bps_limit": "0", 2024-09-22T03:08:13.571 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_qos_exclude_ops": "0", 2024-09-22T03:08:13.571 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_qos_iops_burst": "0", 2024-09-22T03:08:13.571 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_qos_iops_burst_seconds": "1", 2024-09-22T03:08:13.572 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_qos_iops_limit": "0", 2024-09-22T03:08:13.572 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_qos_read_bps_burst": "0", 2024-09-22T03:08:13.572 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_qos_read_bps_burst_seconds": "1", 2024-09-22T03:08:13.572 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_qos_read_bps_limit": "0", 2024-09-22T03:08:13.572 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_qos_read_iops_burst": "0", 2024-09-22T03:08:13.572 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_qos_read_iops_burst_seconds": "1", 2024-09-22T03:08:13.572 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_qos_read_iops_limit": "0", 2024-09-22T03:08:13.572 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_qos_schedule_tick_min": "50", 2024-09-22T03:08:13.572 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_qos_write_bps_burst": "0", 2024-09-22T03:08:13.572 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_qos_write_bps_burst_seconds": "1", 2024-09-22T03:08:13.572 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_qos_write_bps_limit": "0", 2024-09-22T03:08:13.572 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_qos_write_iops_burst": "0", 2024-09-22T03:08:13.572 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_qos_write_iops_burst_seconds": "1", 2024-09-22T03:08:13.573 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_qos_write_iops_limit": "0", 2024-09-22T03:08:13.573 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_quiesce_notification_attempts": "10", 2024-09-22T03:08:13.573 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_read_from_replica_policy": "default", 2024-09-22T03:08:13.573 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_readahead_disable_after_bytes": "52428800", 2024-09-22T03:08:13.573 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_readahead_max_bytes": "524288", 2024-09-22T03:08:13.573 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_readahead_trigger_requests": "10", 2024-09-22T03:08:13.573 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_request_timed_out_seconds": "30", 2024-09-22T03:08:13.573 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_skip_partial_discard": "true", 2024-09-22T03:08:13.573 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_sparse_read_threshold_bytes": "65536", 2024-09-22T03:08:13.573 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_tracing": "false", 2024-09-22T03:08:13.573 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_validate_names": "true", 2024-09-22T03:08:13.573 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_validate_pool": "true", 2024-09-22T03:08:13.573 INFO:teuthology.orchestra.run.smithi153.stdout: "restapi_base_url": "", 2024-09-22T03:08:13.574 INFO:teuthology.orchestra.run.smithi153.stdout: "restapi_log_level": "", 2024-09-22T03:08:13.574 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_account_default_quota_max_objects": "-1", 2024-09-22T03:08:13.574 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_account_default_quota_max_size": "-1", 2024-09-22T03:08:13.574 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_acl_grants_max_num": "100", 2024-09-22T03:08:13.574 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_admin_entry": "admin", 2024-09-22T03:08:13.574 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_allow_notification_secrets_in_cleartext": "false", 2024-09-22T03:08:13.574 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_backend_store": "rados", 2024-09-22T03:08:13.574 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_barbican_url": "", 2024-09-22T03:08:13.574 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_beast_enable_async": "true", 2024-09-22T03:08:13.574 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_bucket_counters_cache": "false", 2024-09-22T03:08:13.574 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_bucket_counters_cache_size": "10000", 2024-09-22T03:08:13.574 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_bucket_default_quota_max_objects": "-1", 2024-09-22T03:08:13.575 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_bucket_default_quota_max_size": "-1", 2024-09-22T03:08:13.575 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_bucket_index_max_aio": "128", 2024-09-22T03:08:13.575 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_bucket_index_transaction_instrumentation": "false", 2024-09-22T03:08:13.575 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_bucket_quota_cache_size": "10000", 2024-09-22T03:08:13.575 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_bucket_quota_ttl": "600", 2024-09-22T03:08:13.575 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_bucket_sync_spawn_window": "20", 2024-09-22T03:08:13.575 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_cache_enabled": "true", 2024-09-22T03:08:13.575 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_cache_expiry_interval": "900", 2024-09-22T03:08:13.575 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_cache_lru_size": "25000", 2024-09-22T03:08:13.575 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_config_store": "rados", 2024-09-22T03:08:13.575 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_content_length_compat": "false", 2024-09-22T03:08:13.575 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_copy_obj_progress": "true", 2024-09-22T03:08:13.575 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_copy_obj_progress_every_bytes": "1048576", 2024-09-22T03:08:13.575 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_cors_rules_max_num": "100", 2024-09-22T03:08:13.576 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_cross_domain_policy": "", 2024-09-22T03:08:13.576 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_crypt_default_encryption_key": "", 2024-09-22T03:08:13.576 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_crypt_kmip_addr": "", 2024-09-22T03:08:13.576 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_crypt_kmip_ca_path": "", 2024-09-22T03:08:13.576 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_crypt_kmip_client_cert": "", 2024-09-22T03:08:13.576 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_crypt_kmip_client_key": "", 2024-09-22T03:08:13.576 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_crypt_kmip_kms_key_template": "", 2024-09-22T03:08:13.576 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_crypt_kmip_password": "", 2024-09-22T03:08:13.576 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_crypt_kmip_s3_key_template": "$keyid", 2024-09-22T03:08:13.576 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_crypt_kmip_username": "", 2024-09-22T03:08:13.576 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_crypt_require_ssl": "true", 2024-09-22T03:08:13.576 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_crypt_s3_kms_backend": "barbican", 2024-09-22T03:08:13.576 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_crypt_s3_kms_encryption_keys": "", 2024-09-22T03:08:13.576 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_crypt_sse_s3_backend": "vault", 2024-09-22T03:08:13.577 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_crypt_sse_s3_key_template": "%bucket_id", 2024-09-22T03:08:13.577 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_crypt_sse_s3_vault_addr": "", 2024-09-22T03:08:13.577 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_crypt_sse_s3_vault_auth": "token", 2024-09-22T03:08:13.577 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_crypt_sse_s3_vault_namespace": "", 2024-09-22T03:08:13.577 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_crypt_sse_s3_vault_prefix": "", 2024-09-22T03:08:13.577 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_crypt_sse_s3_vault_secret_engine": "transit", 2024-09-22T03:08:13.577 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_crypt_sse_s3_vault_ssl_cacert": "", 2024-09-22T03:08:13.577 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_crypt_sse_s3_vault_ssl_clientcert": "", 2024-09-22T03:08:13.577 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_crypt_sse_s3_vault_ssl_clientkey": "", 2024-09-22T03:08:13.577 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_crypt_sse_s3_vault_token_file": "", 2024-09-22T03:08:13.577 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_crypt_sse_s3_vault_verify_ssl": "true", 2024-09-22T03:08:13.577 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_crypt_suppress_logs": "true", 2024-09-22T03:08:13.577 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_crypt_vault_addr": "", 2024-09-22T03:08:13.577 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_crypt_vault_auth": "token", 2024-09-22T03:08:13.578 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_crypt_vault_namespace": "", 2024-09-22T03:08:13.578 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_crypt_vault_prefix": "", 2024-09-22T03:08:13.578 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_crypt_vault_secret_engine": "transit", 2024-09-22T03:08:13.578 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_crypt_vault_ssl_cacert": "", 2024-09-22T03:08:13.578 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_crypt_vault_ssl_clientcert": "", 2024-09-22T03:08:13.578 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_crypt_vault_ssl_clientkey": "", 2024-09-22T03:08:13.578 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_crypt_vault_token_file": "", 2024-09-22T03:08:13.578 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_crypt_vault_verify_ssl": "true", 2024-09-22T03:08:13.578 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_curl_buffersize": "524288", 2024-09-22T03:08:13.578 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_curl_low_speed_limit": "1024", 2024-09-22T03:08:13.578 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_curl_low_speed_time": "300", 2024-09-22T03:08:13.578 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_curl_tcp_keepalive": "0", 2024-09-22T03:08:13.578 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_curl_wait_timeout_ms": "1000", 2024-09-22T03:08:13.579 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_d3n_l1_datacache_persistent_path": "/tmp/rgw_datacache/", 2024-09-22T03:08:13.579 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_d3n_l1_datacache_size": "1073741824", 2024-09-22T03:08:13.579 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_d3n_l1_evict_cache_on_start": "true", 2024-09-22T03:08:13.579 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_d3n_l1_eviction_policy": "lru", 2024-09-22T03:08:13.579 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_d3n_l1_fadvise": "4", 2024-09-22T03:08:13.579 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_d3n_l1_local_datacache_enabled": "false", 2024-09-22T03:08:13.579 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_d3n_libaio_aio_num": "64", 2024-09-22T03:08:13.579 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_d3n_libaio_aio_threads": "20", 2024-09-22T03:08:13.579 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_d4n_host": "127.0.0.1", 2024-09-22T03:08:13.579 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_d4n_port": "6379", 2024-09-22T03:08:13.580 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_data": "/var/lib/ceph/radosgw/ceph-a", 2024-09-22T03:08:13.580 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_data_log_changes_size": "1000", 2024-09-22T03:08:13.580 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_data_log_num_shards": "128", 2024-09-22T03:08:13.580 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_data_log_obj_prefix": "data_log", 2024-09-22T03:08:13.580 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_data_log_window": "30", 2024-09-22T03:08:13.580 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_data_notify_interval_msec": "0", 2024-09-22T03:08:13.580 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_data_sync_poll_interval": "20", 2024-09-22T03:08:13.580 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_data_sync_spawn_window": "20", 2024-09-22T03:08:13.580 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_debug_inject_latency_bi_unlink": "0", 2024-09-22T03:08:13.580 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_debug_inject_olh_cancel_modification_err": "false", 2024-09-22T03:08:13.580 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_debug_inject_set_olh_err": "0", 2024-09-22T03:08:13.580 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_default_data_log_backing": "fifo", 2024-09-22T03:08:13.580 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_default_realm_info_oid": "default.realm", 2024-09-22T03:08:13.581 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_default_region_info_oid": "default.region", 2024-09-22T03:08:13.581 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_default_zone_info_oid": "default.zone", 2024-09-22T03:08:13.581 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_default_zonegroup_info_oid": "default.zonegroup", 2024-09-22T03:08:13.581 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_defer_to_bucket_acls": "", 2024-09-22T03:08:13.581 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_delete_multi_obj_max_num": "1000", 2024-09-22T03:08:13.581 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_dmclock_admin_lim": "0.000000", 2024-09-22T03:08:13.581 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_dmclock_admin_res": "100.000000", 2024-09-22T03:08:13.581 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_dmclock_admin_wgt": "100.000000", 2024-09-22T03:08:13.581 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_dmclock_auth_lim": "0.000000", 2024-09-22T03:08:13.581 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_dmclock_auth_res": "200.000000", 2024-09-22T03:08:13.581 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_dmclock_auth_wgt": "100.000000", 2024-09-22T03:08:13.581 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_dmclock_data_lim": "0.000000", 2024-09-22T03:08:13.581 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_dmclock_data_res": "500.000000", 2024-09-22T03:08:13.581 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_dmclock_data_wgt": "500.000000", 2024-09-22T03:08:13.582 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_dmclock_metadata_lim": "0.000000", 2024-09-22T03:08:13.582 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_dmclock_metadata_res": "500.000000", 2024-09-22T03:08:13.582 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_dmclock_metadata_wgt": "500.000000", 2024-09-22T03:08:13.582 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_dns_name": "", 2024-09-22T03:08:13.582 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_dns_s3website_name": "", 2024-09-22T03:08:13.582 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_dynamic_resharding": "true", 2024-09-22T03:08:13.582 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_enable_apis": "s3, s3website, swift, swift_auth, admin, sts, iam, notifications", 2024-09-22T03:08:13.582 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_enable_gc_threads": "true", 2024-09-22T03:08:13.582 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_enable_lc_threads": "true", 2024-09-22T03:08:13.582 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_enable_ops_log": "false", 2024-09-22T03:08:13.582 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_enable_quota_threads": "true", 2024-09-22T03:08:13.582 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_enable_static_website": "false", 2024-09-22T03:08:13.582 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_enable_usage_log": "false", 2024-09-22T03:08:13.583 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_enforce_swift_acls": "true", 2024-09-22T03:08:13.583 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_exit_timeout_secs": "120", 2024-09-22T03:08:13.583 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_expose_bucket": "false", 2024-09-22T03:08:13.583 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_extended_http_attrs": "", 2024-09-22T03:08:13.583 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_filter": "none", 2024-09-22T03:08:13.583 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_frontend_defaults": "beast ssl_certificate=config://rgw/cert/$realm/$zone.crt ssl_private_key=config://rgw/cert/$realm/$zone.key", 2024-09-22T03:08:13.583 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_frontends": "beast port=7480", 2024-09-22T03:08:13.583 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_gc_max_concurrent_io": "10", 2024-09-22T03:08:13.583 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_gc_max_deferred": "50", 2024-09-22T03:08:13.583 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_gc_max_deferred_entries_size": "3072", 2024-09-22T03:08:13.583 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_gc_max_objs": "32", 2024-09-22T03:08:13.583 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_gc_max_queue_size": "134213632", 2024-09-22T03:08:13.583 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_gc_max_trim_chunk": "16", 2024-09-22T03:08:13.583 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_gc_obj_min_wait": "7200", 2024-09-22T03:08:13.584 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_gc_processor_max_time": "3600", 2024-09-22T03:08:13.584 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_gc_processor_period": "3600", 2024-09-22T03:08:13.584 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_get_obj_max_req_size": "4194304", 2024-09-22T03:08:13.584 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_get_obj_window_size": "16777216", 2024-09-22T03:08:13.584 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_healthcheck_disabling_path": "", 2024-09-22T03:08:13.584 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_ignore_get_invalid_range": "false", 2024-09-22T03:08:13.584 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_init_timeout": "300", 2024-09-22T03:08:13.584 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_inject_notify_timeout_probability": "0.000000", 2024-09-22T03:08:13.584 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_json_config": "/var/lib/ceph/radosgw/config.json", 2024-09-22T03:08:13.584 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_kafka_connection_idle": "300", 2024-09-22T03:08:13.584 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_kafka_message_timeout": "5000", 2024-09-22T03:08:13.584 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_kafka_sleep_timeout": "10", 2024-09-22T03:08:13.584 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_keystone_accepted_admin_roles": "", 2024-09-22T03:08:13.584 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_keystone_accepted_reader_roles": "", 2024-09-22T03:08:13.584 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_keystone_accepted_roles": "Member, admin", 2024-09-22T03:08:13.585 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_keystone_admin_domain": "", 2024-09-22T03:08:13.585 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_keystone_admin_password": "", 2024-09-22T03:08:13.585 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_keystone_admin_password_path": "", 2024-09-22T03:08:13.585 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_keystone_admin_project": "", 2024-09-22T03:08:13.585 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_keystone_admin_tenant": "", 2024-09-22T03:08:13.585 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_keystone_admin_token": "", 2024-09-22T03:08:13.585 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_keystone_admin_token_path": "", 2024-09-22T03:08:13.585 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_keystone_admin_user": "", 2024-09-22T03:08:13.585 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_keystone_api_version": "2", 2024-09-22T03:08:13.585 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_keystone_barbican_domain": "", 2024-09-22T03:08:13.585 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_keystone_barbican_password": "", 2024-09-22T03:08:13.585 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_keystone_barbican_project": "", 2024-09-22T03:08:13.585 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_keystone_barbican_tenant": "", 2024-09-22T03:08:13.585 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_keystone_barbican_user": "", 2024-09-22T03:08:13.586 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_keystone_expired_token_cache_expiration": "3600", 2024-09-22T03:08:13.586 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_keystone_implicit_tenants": "false", 2024-09-22T03:08:13.586 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_keystone_service_token_accepted_roles": "admin", 2024-09-22T03:08:13.586 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_keystone_service_token_enabled": "false", 2024-09-22T03:08:13.586 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_keystone_token_cache_size": "10000", 2024-09-22T03:08:13.586 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_keystone_url": "", 2024-09-22T03:08:13.586 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_keystone_verify_ssl": "true", 2024-09-22T03:08:13.586 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_lc_debug_interval": "-1", 2024-09-22T03:08:13.586 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_lc_lock_max_time": "90", 2024-09-22T03:08:13.586 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_lc_max_objs": "32", 2024-09-22T03:08:13.586 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_lc_max_rules": "1000", 2024-09-22T03:08:13.586 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_lc_max_worker": "3", 2024-09-22T03:08:13.586 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_lc_max_wp_worker": "3", 2024-09-22T03:08:13.586 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_lc_thread_delay": "0", 2024-09-22T03:08:13.587 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_ldap_binddn": "uid=admin,cn=users,dc=example,dc=com", 2024-09-22T03:08:13.587 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_ldap_dnattr": "uid", 2024-09-22T03:08:13.587 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_ldap_searchdn": "cn=users,cn=accounts,dc=example,dc=com", 2024-09-22T03:08:13.587 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_ldap_searchfilter": "", 2024-09-22T03:08:13.587 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_ldap_secret": "/etc/openldap/secret", 2024-09-22T03:08:13.587 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_ldap_uri": "", 2024-09-22T03:08:13.587 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_lifecycle_work_time": "00:00-06:00", 2024-09-22T03:08:13.587 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_list_bucket_min_readahead": "1000", 2024-09-22T03:08:13.587 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_list_buckets_max_chunk": "1000", 2024-09-22T03:08:13.587 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_log_http_headers": "", 2024-09-22T03:08:13.587 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_log_nonexistent_bucket": "false", 2024-09-22T03:08:13.587 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_log_object_name": "%Y-%m-%d-%H-%i-%n", 2024-09-22T03:08:13.587 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_log_object_name_utc": "false", 2024-09-22T03:08:13.587 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_lua_max_memory_per_state": "128000", 2024-09-22T03:08:13.588 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_luarocks_location": "/tmp/rgw_luarocks/mon.a", 2024-09-22T03:08:13.588 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_max_attr_name_len": "0", 2024-09-22T03:08:13.588 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_max_attr_size": "0", 2024-09-22T03:08:13.588 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_max_attrs_num_in_req": "0", 2024-09-22T03:08:13.588 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_max_chunk_size": "4194304", 2024-09-22T03:08:13.588 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_max_concurrent_requests": "1024", 2024-09-22T03:08:13.588 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_max_copy_obj_concurrent_io": "10", 2024-09-22T03:08:13.588 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_max_dynamic_shards": "1999", 2024-09-22T03:08:13.588 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_max_listing_results": "1000", 2024-09-22T03:08:13.588 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_max_notify_retries": "10", 2024-09-22T03:08:13.588 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_max_objs_per_shard": "100000", 2024-09-22T03:08:13.588 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_max_put_param_size": "1048576", 2024-09-22T03:08:13.588 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_max_put_size": "5368709120", 2024-09-22T03:08:13.589 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_max_slo_entries": "1000", 2024-09-22T03:08:13.589 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_md_log_max_shards": "64", 2024-09-22T03:08:13.589 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_md_notify_interval_msec": "200", 2024-09-22T03:08:13.589 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_meta_sync_poll_interval": "20", 2024-09-22T03:08:13.589 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_meta_sync_spawn_window": "20", 2024-09-22T03:08:13.589 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_mime_types_file": "/etc/mime.types", 2024-09-22T03:08:13.589 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_mp_lock_max_time": "600", 2024-09-22T03:08:13.589 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_multi_obj_del_max_aio": "16", 2024-09-22T03:08:13.589 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_multipart_min_part_size": "5242880", 2024-09-22T03:08:13.589 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_multipart_part_upload_limit": "10000", 2024-09-22T03:08:13.589 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_nfs_fhcache_partitions": "3", 2024-09-22T03:08:13.589 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_nfs_fhcache_size": "2017", 2024-09-22T03:08:13.589 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_nfs_frontends": "rgw-nfs", 2024-09-22T03:08:13.589 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_nfs_lru_lane_hiwat": "911", 2024-09-22T03:08:13.590 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_nfs_lru_lanes": "5", 2024-09-22T03:08:13.590 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_nfs_max_gc": "300", 2024-09-22T03:08:13.590 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_nfs_namespace_expire_secs": "300", 2024-09-22T03:08:13.590 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_nfs_run_gc_threads": "false", 2024-09-22T03:08:13.590 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_nfs_run_lc_threads": "false", 2024-09-22T03:08:13.590 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_nfs_run_quota_threads": "false", 2024-09-22T03:08:13.590 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_nfs_run_sync_thread": "false", 2024-09-22T03:08:13.590 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_nfs_s3_fast_attrs": "false", 2024-09-22T03:08:13.590 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_nfs_write_completion_interval_s": "10", 2024-09-22T03:08:13.590 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_num_async_rados_threads": "32", 2024-09-22T03:08:13.590 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_num_control_oids": "8", 2024-09-22T03:08:13.590 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_numa_node": "-1", 2024-09-22T03:08:13.590 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_obj_stripe_size": "4194304", 2024-09-22T03:08:13.590 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_obj_tombstone_cache_size": "1000", 2024-09-22T03:08:13.591 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_objexp_chunk_size": "100", 2024-09-22T03:08:13.591 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_objexp_gc_interval": "600", 2024-09-22T03:08:13.591 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_objexp_hints_num_shards": "127", 2024-09-22T03:08:13.591 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_olh_pending_timeout_sec": "3600", 2024-09-22T03:08:13.591 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_op_thread_suicide_timeout": "0", 2024-09-22T03:08:13.591 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_op_thread_timeout": "600", 2024-09-22T03:08:13.591 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_op_tracing": "false", 2024-09-22T03:08:13.591 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_opa_token": "", 2024-09-22T03:08:13.591 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_opa_url": "", 2024-09-22T03:08:13.591 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_opa_verify_ssl": "true", 2024-09-22T03:08:13.591 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_ops_log_data_backlog": "5242880", 2024-09-22T03:08:13.591 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_ops_log_file_path": "/var/log/ceph/ops-log-ceph-mon.a.log", 2024-09-22T03:08:13.591 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_ops_log_rados": "false", 2024-09-22T03:08:13.592 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_ops_log_socket_path": "", 2024-09-22T03:08:13.592 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_override_bucket_index_max_shards": "0", 2024-09-22T03:08:13.592 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_pending_bucket_index_op_expiration": "120", 2024-09-22T03:08:13.592 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_period_latest_epoch_info_oid": ".latest_epoch", 2024-09-22T03:08:13.592 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_period_push_interval": "2.000000", 2024-09-22T03:08:13.592 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_period_push_interval_max": "30.000000", 2024-09-22T03:08:13.592 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_period_root_pool": ".rgw.root", 2024-09-22T03:08:13.592 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_policy_reject_invalid_principals": "true", 2024-09-22T03:08:13.592 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_posix_base_path": "/tmp/rgw_posix_driver", 2024-09-22T03:08:13.592 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_posix_cache_lanes": "3", 2024-09-22T03:08:13.592 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_posix_cache_lmdb_count": "3", 2024-09-22T03:08:13.592 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_posix_cache_max_buckets": "100", 2024-09-22T03:08:13.592 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_posix_cache_partitions": "3", 2024-09-22T03:08:13.593 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_posix_database_root": "/var/lib/ceph/radosgw", 2024-09-22T03:08:13.593 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_print_continue": "true", 2024-09-22T03:08:13.593 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_print_prohibited_content_length": "false", 2024-09-22T03:08:13.593 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_put_obj_max_window_size": "67108864", 2024-09-22T03:08:13.593 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_put_obj_min_window_size": "16777216", 2024-09-22T03:08:13.593 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_rados_pool_autoscale_bias": "4.000000", 2024-09-22T03:08:13.593 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_rados_pool_recovery_priority": "5", 2024-09-22T03:08:13.593 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_rados_tracing": "false", 2024-09-22T03:08:13.593 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_realm": "", 2024-09-22T03:08:13.593 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_realm_id": "", 2024-09-22T03:08:13.593 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_realm_root_pool": ".rgw.root", 2024-09-22T03:08:13.593 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_region": "", 2024-09-22T03:08:13.593 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_region_root_pool": ".rgw.root", 2024-09-22T03:08:13.594 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_relaxed_region_enforcement": "false", 2024-09-22T03:08:13.594 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_relaxed_s3_bucket_names": "false", 2024-09-22T03:08:13.594 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_remote_addr_param": "REMOTE_ADDR", 2024-09-22T03:08:13.594 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_request_uri": "", 2024-09-22T03:08:13.594 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_reshard_batch_size": "64", 2024-09-22T03:08:13.594 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_reshard_bucket_lock_duration": "360", 2024-09-22T03:08:13.594 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_reshard_max_aio": "128", 2024-09-22T03:08:13.594 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_reshard_num_logs": "16", 2024-09-22T03:08:13.594 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_reshard_thread_interval": "600", 2024-09-22T03:08:13.594 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_resolve_cname": "false", 2024-09-22T03:08:13.594 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_rest_getusage_op_compat": "false", 2024-09-22T03:08:13.594 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_run_sync_thread": "true", 2024-09-22T03:08:13.594 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_s3_auth_disable_signature_url": "false", 2024-09-22T03:08:13.594 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_s3_auth_order": "sts, external, local", 2024-09-22T03:08:13.595 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_s3_auth_use_keystone": "false", 2024-09-22T03:08:13.595 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_s3_auth_use_ldap": "false", 2024-09-22T03:08:13.595 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_s3_auth_use_rados": "true", 2024-09-22T03:08:13.595 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_s3_auth_use_sts": "false", 2024-09-22T03:08:13.595 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_s3_client_max_sig_ver": "-1", 2024-09-22T03:08:13.595 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_s3_success_create_obj_status": "0", 2024-09-22T03:08:13.595 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_safe_max_objects_per_shard": "102400", 2024-09-22T03:08:13.595 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_scheduler_type": "throttler", 2024-09-22T03:08:13.595 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_script_uri": "", 2024-09-22T03:08:13.595 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_service_provider_name": "", 2024-09-22T03:08:13.595 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_shard_warning_threshold": "90.000000", 2024-09-22T03:08:13.595 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_sts_client_id": "", 2024-09-22T03:08:13.595 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_sts_client_secret": "", 2024-09-22T03:08:13.596 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_sts_entry": "sts", 2024-09-22T03:08:13.596 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_sts_key": "sts", 2024-09-22T03:08:13.596 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_sts_max_session_duration": "43200", 2024-09-22T03:08:13.596 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_sts_min_session_duration": "900", 2024-09-22T03:08:13.596 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_sts_token_introspection_url": "", 2024-09-22T03:08:13.596 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_swift_account_in_url": "false", 2024-09-22T03:08:13.596 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_swift_auth_entry": "auth", 2024-09-22T03:08:13.596 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_swift_auth_url": "", 2024-09-22T03:08:13.596 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_swift_custom_header": "", 2024-09-22T03:08:13.596 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_swift_enforce_content_length": "false", 2024-09-22T03:08:13.596 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_swift_need_stats": "true", 2024-09-22T03:08:13.596 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_swift_tenant_name": "", 2024-09-22T03:08:13.596 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_swift_token_expiration": "86400", 2024-09-22T03:08:13.596 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_swift_url": "", 2024-09-22T03:08:13.596 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_swift_url_prefix": "swift", 2024-09-22T03:08:13.597 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_swift_versioning_enabled": "false", 2024-09-22T03:08:13.597 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_sync_data_full_inject_err_probability": "0.000000", 2024-09-22T03:08:13.597 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_sync_data_inject_err_probability": "0.000000", 2024-09-22T03:08:13.597 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_sync_lease_period": "120", 2024-09-22T03:08:13.597 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_sync_log_trim_concurrent_buckets": "4", 2024-09-22T03:08:13.597 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_sync_log_trim_interval": "1200", 2024-09-22T03:08:13.597 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_sync_log_trim_max_buckets": "16", 2024-09-22T03:08:13.597 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_sync_log_trim_min_cold_buckets": "4", 2024-09-22T03:08:13.597 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_sync_meta_inject_err_probability": "0.000000", 2024-09-22T03:08:13.597 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_sync_obj_etag_verify": "false", 2024-09-22T03:08:13.597 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_sync_trace_history_size": "4096", 2024-09-22T03:08:13.597 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_sync_trace_per_node_log_size": "32", 2024-09-22T03:08:13.597 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_sync_trace_servicemap_update_interval": "10", 2024-09-22T03:08:13.597 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_thread_pool_size": "512", 2024-09-22T03:08:13.598 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_topic_persistency_max_retries": "0", 2024-09-22T03:08:13.598 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_topic_persistency_sleep_duration": "0", 2024-09-22T03:08:13.598 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_topic_persistency_time_to_live": "0", 2024-09-22T03:08:13.598 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_topic_require_publish_policy": "false", 2024-09-22T03:08:13.598 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_torrent_comment": "", 2024-09-22T03:08:13.598 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_torrent_createby": "", 2024-09-22T03:08:13.598 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_torrent_encoding": "", 2024-09-22T03:08:13.598 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_torrent_flag": "false", 2024-09-22T03:08:13.598 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_torrent_max_size": "5368709120", 2024-09-22T03:08:13.598 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_torrent_origin": "", 2024-09-22T03:08:13.598 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_torrent_sha_unit": "524288", 2024-09-22T03:08:13.598 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_torrent_tracker": "", 2024-09-22T03:08:13.598 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_trust_forwarded_https": "false", 2024-09-22T03:08:13.598 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_usage_log_flush_threshold": "1024", 2024-09-22T03:08:13.599 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_usage_log_tick_interval": "30", 2024-09-22T03:08:13.599 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_usage_max_shards": "32", 2024-09-22T03:08:13.599 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_usage_max_user_shards": "1", 2024-09-22T03:08:13.599 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_use_opa_authz": "false", 2024-09-22T03:08:13.599 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_user_counters_cache": "false", 2024-09-22T03:08:13.599 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_user_counters_cache_size": "10000", 2024-09-22T03:08:13.599 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_user_default_quota_max_objects": "-1", 2024-09-22T03:08:13.599 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_user_default_quota_max_size": "-1", 2024-09-22T03:08:13.599 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_user_max_buckets": "1000", 2024-09-22T03:08:13.599 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_user_policies_max_num": "100", 2024-09-22T03:08:13.599 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_user_quota_bucket_sync_interval": "180", 2024-09-22T03:08:13.599 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_user_quota_sync_idle_users": "false", 2024-09-22T03:08:13.599 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_user_quota_sync_interval": "86400", 2024-09-22T03:08:13.599 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_user_quota_sync_wait_time": "86400", 2024-09-22T03:08:13.600 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_user_unique_email": "true", 2024-09-22T03:08:13.600 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_verify_ssl": "true", 2024-09-22T03:08:13.600 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_website_routing_rules_max_num": "50", 2024-09-22T03:08:13.600 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_zone": "", 2024-09-22T03:08:13.600 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_zone_id": "", 2024-09-22T03:08:13.600 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_zone_root_pool": ".rgw.root", 2024-09-22T03:08:13.600 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_zonegroup": "", 2024-09-22T03:08:13.600 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_zonegroup_id": "", 2024-09-22T03:08:13.600 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_zonegroup_root_pool": ".rgw.root", 2024-09-22T03:08:13.600 INFO:teuthology.orchestra.run.smithi153.stdout: "rgwlc_auto_session_clear": "true", 2024-09-22T03:08:13.600 INFO:teuthology.orchestra.run.smithi153.stdout: "rgwlc_skip_bucket_step": "false", 2024-09-22T03:08:13.600 INFO:teuthology.orchestra.run.smithi153.stdout: "rocksdb_block_size": "4096", 2024-09-22T03:08:13.600 INFO:teuthology.orchestra.run.smithi153.stdout: "rocksdb_bloom_bits_per_key": "20", 2024-09-22T03:08:13.600 INFO:teuthology.orchestra.run.smithi153.stdout: "rocksdb_cache_index_and_filter_blocks": "true", 2024-09-22T03:08:13.600 INFO:teuthology.orchestra.run.smithi153.stdout: "rocksdb_cache_index_and_filter_blocks_with_high_priority": "false", 2024-09-22T03:08:13.601 INFO:teuthology.orchestra.run.smithi153.stdout: "rocksdb_cache_row_ratio": "0.000000", 2024-09-22T03:08:13.601 INFO:teuthology.orchestra.run.smithi153.stdout: "rocksdb_cache_shard_bits": "4", 2024-09-22T03:08:13.601 INFO:teuthology.orchestra.run.smithi153.stdout: "rocksdb_cache_size": "536870912", 2024-09-22T03:08:13.601 INFO:teuthology.orchestra.run.smithi153.stdout: "rocksdb_cache_type": "binned_lru", 2024-09-22T03:08:13.601 INFO:teuthology.orchestra.run.smithi153.stdout: "rocksdb_cf_compact_on_deletion": "true", 2024-09-22T03:08:13.601 INFO:teuthology.orchestra.run.smithi153.stdout: "rocksdb_cf_compact_on_deletion_sliding_window": "32768", 2024-09-22T03:08:13.601 INFO:teuthology.orchestra.run.smithi153.stdout: "rocksdb_cf_compact_on_deletion_trigger": "16384", 2024-09-22T03:08:13.601 INFO:teuthology.orchestra.run.smithi153.stdout: "rocksdb_collect_compaction_stats": "false", 2024-09-22T03:08:13.601 INFO:teuthology.orchestra.run.smithi153.stdout: "rocksdb_collect_extended_stats": "false", 2024-09-22T03:08:13.601 INFO:teuthology.orchestra.run.smithi153.stdout: "rocksdb_collect_memory_stats": "false", 2024-09-22T03:08:13.601 INFO:teuthology.orchestra.run.smithi153.stdout: "rocksdb_delete_range_threshold": "1048576", 2024-09-22T03:08:13.601 INFO:teuthology.orchestra.run.smithi153.stdout: "rocksdb_index_type": "binary_search", 2024-09-22T03:08:13.601 INFO:teuthology.orchestra.run.smithi153.stdout: "rocksdb_log_to_ceph_log": "true", 2024-09-22T03:08:13.602 INFO:teuthology.orchestra.run.smithi153.stdout: "rocksdb_metadata_block_size": "4096", 2024-09-22T03:08:13.602 INFO:teuthology.orchestra.run.smithi153.stdout: "rocksdb_partition_filters": "false", 2024-09-22T03:08:13.602 INFO:teuthology.orchestra.run.smithi153.stdout: "rocksdb_perf": "false", 2024-09-22T03:08:13.602 INFO:teuthology.orchestra.run.smithi153.stdout: "rocksdb_pin_l0_filter_and_index_blocks_in_cache": "false", 2024-09-22T03:08:13.602 INFO:teuthology.orchestra.run.smithi153.stdout: "rotating_keys_bootstrap_timeout": "30", 2024-09-22T03:08:13.602 INFO:teuthology.orchestra.run.smithi153.stdout: "rotating_keys_renewal_timeout": "10", 2024-09-22T03:08:13.602 INFO:teuthology.orchestra.run.smithi153.stdout: "run_dir": "/var/run/ceph", 2024-09-22T03:08:13.602 INFO:teuthology.orchestra.run.smithi153.stdout: "seastore_block_create": "true", 2024-09-22T03:08:13.602 INFO:teuthology.orchestra.run.smithi153.stdout: "seastore_cache_lru_size": "67108864", 2024-09-22T03:08:13.602 INFO:teuthology.orchestra.run.smithi153.stdout: "seastore_cbjournal_size": "5368709120", 2024-09-22T03:08:13.602 INFO:teuthology.orchestra.run.smithi153.stdout: "seastore_data_delta_based_overwrite": "0", 2024-09-22T03:08:13.602 INFO:teuthology.orchestra.run.smithi153.stdout: "seastore_default_max_object_size": "16777216", 2024-09-22T03:08:13.603 INFO:teuthology.orchestra.run.smithi153.stdout: "seastore_default_object_metadata_reservation": "16777216", 2024-09-22T03:08:13.603 INFO:teuthology.orchestra.run.smithi153.stdout: "seastore_device_size": "53687091200", 2024-09-22T03:08:13.603 INFO:teuthology.orchestra.run.smithi153.stdout: "seastore_disable_end_to_end_data_protection": "true", 2024-09-22T03:08:13.603 INFO:teuthology.orchestra.run.smithi153.stdout: "seastore_full_integrity_check": "false", 2024-09-22T03:08:13.603 INFO:teuthology.orchestra.run.smithi153.stdout: "seastore_journal_batch_capacity": "16", 2024-09-22T03:08:13.603 INFO:teuthology.orchestra.run.smithi153.stdout: "seastore_journal_batch_flush_size": "16777216", 2024-09-22T03:08:13.603 INFO:teuthology.orchestra.run.smithi153.stdout: "seastore_journal_batch_preferred_fullness": "0.950000", 2024-09-22T03:08:13.603 INFO:teuthology.orchestra.run.smithi153.stdout: "seastore_journal_iodepth_limit": "5", 2024-09-22T03:08:13.603 INFO:teuthology.orchestra.run.smithi153.stdout: "seastore_main_device_type": "SSD", 2024-09-22T03:08:13.603 INFO:teuthology.orchestra.run.smithi153.stdout: "seastore_max_concurrent_transactions": "8", 2024-09-22T03:08:13.603 INFO:teuthology.orchestra.run.smithi153.stdout: "seastore_max_data_allocation_size": "32768", 2024-09-22T03:08:13.603 INFO:teuthology.orchestra.run.smithi153.stdout: "seastore_multiple_tiers_default_evict_ratio": "0.600000", 2024-09-22T03:08:13.603 INFO:teuthology.orchestra.run.smithi153.stdout: "seastore_multiple_tiers_fast_evict_ratio": "0.700000", 2024-09-22T03:08:13.604 INFO:teuthology.orchestra.run.smithi153.stdout: "seastore_multiple_tiers_stop_evict_ratio": "0.500000", 2024-09-22T03:08:13.604 INFO:teuthology.orchestra.run.smithi153.stdout: "seastore_obj_data_write_amplification": "1.250000", 2024-09-22T03:08:13.604 INFO:teuthology.orchestra.run.smithi153.stdout: "seastore_segment_size": "67108864", 2024-09-22T03:08:13.604 INFO:teuthology.orchestra.run.smithi153.stdout: "set_keepcaps": "false", 2024-09-22T03:08:13.604 INFO:teuthology.orchestra.run.smithi153.stdout: "setgroup": "", 2024-09-22T03:08:13.604 INFO:teuthology.orchestra.run.smithi153.stdout: "setuser": "", 2024-09-22T03:08:13.604 INFO:teuthology.orchestra.run.smithi153.stdout: "setuser_match_path": "", 2024-09-22T03:08:13.604 INFO:teuthology.orchestra.run.smithi153.stdout: "target_max_misplaced_ratio": "0.050000", 2024-09-22T03:08:13.604 INFO:teuthology.orchestra.run.smithi153.stdout: "thp": "false", 2024-09-22T03:08:13.604 INFO:teuthology.orchestra.run.smithi153.stdout: "threadpool_default_timeout": "60", 2024-09-22T03:08:13.604 INFO:teuthology.orchestra.run.smithi153.stdout: "threadpool_empty_queue_max_wait": "2", 2024-09-22T03:08:13.604 INFO:teuthology.orchestra.run.smithi153.stdout: "throttler_perf_counter": "true" 2024-09-22T03:08:13.604 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-22T03:08:13.606 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=0 2024-09-22T03:08:13.709 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:08:13.706+0000 7f37acd17640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:08:13.712 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:08:13.710+0000 7f37acd17640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:08:13.724 INFO:teuthology.orchestra.run.smithi153.stdout:{} 2024-09-22T03:08:13.725 INFO:teuthology.orchestra.run.smithi153.stderr:mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-22T03:08:13.735 DEBUG:teuthology.orchestra.run.smithi153:> type ceph-objectstore-tool 2024-09-22T03:08:13.749 INFO:teuthology.orchestra.run.smithi153.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-09-22T03:08:13.749 DEBUG:teuthology.orchestra.run.smithi196:> type ceph-objectstore-tool 2024-09-22T03:08:13.768 INFO:teuthology.orchestra.run.smithi196.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-09-22T03:08:13.768 INFO:teuthology.run_tasks:Running task full_sequential... 2024-09-22T03:08:13.777 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-09-22T03:08:13.778 INFO:tasks.radosbench:Beginning radosbench... 2024-09-22T03:08:13.778 INFO:tasks.ceph.ceph_manager.ceph:creating pool_name unique_pool_0 2024-09-22T03:08:13.779 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create unique_pool_0 16 2024-09-22T03:08:13.779 INFO:tasks.thrashosds.thrasher:starting do_thrash 2024-09-22T03:08:13.779 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-09-22T03:08:13.779 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-22T03:08:13.779 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-09-22T03:08:13.779 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-09-22T03:08:13.780 INFO:tasks.thrashosds.thrasher:starting do_optrack_toggle with a delay of 2.0 2024-09-22T03:08:13.780 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-22T03:08:13.898 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:08:13.895+0000 7f19d406c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:08:13.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:08:13.895+0000 7f5d84863640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:08:13.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:08:13.900+0000 7f19d406c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:08:13.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:08:13.895+0000 7f0125997640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:08:13.905 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:08:13.904+0000 7f5d84863640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:08:13.908 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:08:13.906+0000 7f0125997640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:08:13.922 INFO:teuthology.orchestra.run.smithi153.stderr:no valid command found; 10 closest matches: 2024-09-22T03:08:13.923 INFO:teuthology.orchestra.run.smithi153.stderr:config get 2024-09-22T03:08:13.923 INFO:teuthology.orchestra.run.smithi153.stderr:config help 2024-09-22T03:08:13.923 INFO:teuthology.orchestra.run.smithi153.stderr:config set ... 2024-09-22T03:08:13.923 INFO:teuthology.orchestra.run.smithi153.stderr:config show 2024-09-22T03:08:13.923 INFO:teuthology.orchestra.run.smithi153.stderr:deep_scrub 2024-09-22T03:08:13.923 INFO:teuthology.orchestra.run.smithi153.stderr:dump_historic_ops 2024-09-22T03:08:13.923 INFO:teuthology.orchestra.run.smithi153.stderr:dump_historic_slow_ops 2024-09-22T03:08:13.923 INFO:teuthology.orchestra.run.smithi153.stderr:dump_metrics [] 2024-09-22T03:08:13.923 INFO:teuthology.orchestra.run.smithi153.stderr:dump_ops_in_flight 2024-09-22T03:08:13.923 INFO:teuthology.orchestra.run.smithi153.stderr:dump_pgstate_history 2024-09-22T03:08:13.923 INFO:teuthology.orchestra.run.smithi153.stderr:Error EINVAL: invalid command 2024-09-22T03:08:13.926 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-22T03:08:13.927 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-22T03:08:13.927 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-09-22T03:08:14.036 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:08:14.030+0000 7f407e557640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:08:14.039 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:08:14.037+0000 7f407e557640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:08:14.053 INFO:teuthology.orchestra.run.smithi153.stderr:no valid command found; 10 closest matches: 2024-09-22T03:08:14.053 INFO:teuthology.orchestra.run.smithi153.stderr:config get 2024-09-22T03:08:14.053 INFO:teuthology.orchestra.run.smithi153.stderr:config help 2024-09-22T03:08:14.053 INFO:teuthology.orchestra.run.smithi153.stderr:config set ... 2024-09-22T03:08:14.053 INFO:teuthology.orchestra.run.smithi153.stderr:config show 2024-09-22T03:08:14.053 INFO:teuthology.orchestra.run.smithi153.stderr:deep_scrub 2024-09-22T03:08:14.053 INFO:teuthology.orchestra.run.smithi153.stderr:dump_historic_ops 2024-09-22T03:08:14.053 INFO:teuthology.orchestra.run.smithi153.stderr:dump_historic_slow_ops 2024-09-22T03:08:14.053 INFO:teuthology.orchestra.run.smithi153.stderr:dump_metrics [] 2024-09-22T03:08:14.053 INFO:teuthology.orchestra.run.smithi153.stderr:dump_ops_in_flight 2024-09-22T03:08:14.054 INFO:teuthology.orchestra.run.smithi153.stderr:dump_pgstate_history 2024-09-22T03:08:14.054 INFO:teuthology.orchestra.run.smithi153.stderr:Error EINVAL: invalid command 2024-09-22T03:08:14.056 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-22T03:08:14.056 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-22T03:08:14.056 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-09-22T03:08:14.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:08:14.157+0000 7f1fb5e6c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:08:14.166 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:08:14.164+0000 7f1fb5e6c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:08:14.180 INFO:teuthology.orchestra.run.smithi153.stderr:no valid command found; 10 closest matches: 2024-09-22T03:08:14.180 INFO:teuthology.orchestra.run.smithi153.stderr:config get 2024-09-22T03:08:14.180 INFO:teuthology.orchestra.run.smithi153.stderr:config help 2024-09-22T03:08:14.180 INFO:teuthology.orchestra.run.smithi153.stderr:config set ... 2024-09-22T03:08:14.180 INFO:teuthology.orchestra.run.smithi153.stderr:config show 2024-09-22T03:08:14.180 INFO:teuthology.orchestra.run.smithi153.stderr:deep_scrub 2024-09-22T03:08:14.180 INFO:teuthology.orchestra.run.smithi153.stderr:dump_historic_ops 2024-09-22T03:08:14.180 INFO:teuthology.orchestra.run.smithi153.stderr:dump_historic_slow_ops 2024-09-22T03:08:14.180 INFO:teuthology.orchestra.run.smithi153.stderr:dump_metrics [] 2024-09-22T03:08:14.180 INFO:teuthology.orchestra.run.smithi153.stderr:dump_ops_in_flight 2024-09-22T03:08:14.180 INFO:teuthology.orchestra.run.smithi153.stderr:dump_pgstate_history 2024-09-22T03:08:14.180 INFO:teuthology.orchestra.run.smithi153.stderr:Error EINVAL: invalid command 2024-09-22T03:08:14.182 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-22T03:08:14.183 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-22T03:08:14.183 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-09-22T03:08:14.287 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:08:14.281+0000 7f91f810e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:08:14.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:08:14.289+0000 7f91f810e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:08:14.303 INFO:teuthology.orchestra.run.smithi153.stderr:no valid command found; 10 closest matches: 2024-09-22T03:08:14.303 INFO:teuthology.orchestra.run.smithi153.stderr:config get 2024-09-22T03:08:14.304 INFO:teuthology.orchestra.run.smithi153.stderr:config help 2024-09-22T03:08:14.304 INFO:teuthology.orchestra.run.smithi153.stderr:config set ... 2024-09-22T03:08:14.304 INFO:teuthology.orchestra.run.smithi153.stderr:config show 2024-09-22T03:08:14.304 INFO:teuthology.orchestra.run.smithi153.stderr:deep_scrub 2024-09-22T03:08:14.304 INFO:teuthology.orchestra.run.smithi153.stderr:dump_historic_ops 2024-09-22T03:08:14.304 INFO:teuthology.orchestra.run.smithi153.stderr:dump_historic_slow_ops 2024-09-22T03:08:14.304 INFO:teuthology.orchestra.run.smithi153.stderr:dump_metrics [] 2024-09-22T03:08:14.304 INFO:teuthology.orchestra.run.smithi153.stderr:dump_ops_in_flight 2024-09-22T03:08:14.304 INFO:teuthology.orchestra.run.smithi153.stderr:dump_pgstate_history 2024-09-22T03:08:14.304 INFO:teuthology.orchestra.run.smithi153.stderr:Error EINVAL: invalid command 2024-09-22T03:08:14.306 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-22T03:08:14.307 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-22T03:08:14.443 INFO:teuthology.orchestra.run.smithi153.stderr:pool 'unique_pool_0' created 2024-09-22T03:08:14.459 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool application enable unique_pool_0 rados --yes-i-really-mean-it || true 2024-09-22T03:08:14.579 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:08:14.569+0000 7f43a1197640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:08:14.582 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:08:14.581+0000 7f43a1197640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:08:15.450 INFO:teuthology.orchestra.run.smithi153.stderr:enabled application 'rados' on pool 'unique_pool_0' 2024-09-22T03:08:15.985 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-22T03:08:16.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:08:16.086+0000 7fe11bb97640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:08:16.092 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:08:16.091+0000 7fe11bb97640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:08:16.475 DEBUG:teuthology.orchestra.run.smithi153:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 8192 --object-size 65536 -p unique_pool_0 bench 90 write' 2024-09-22T03:08:16.478 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-09-22T03:08:16.478 INFO:teuthology.orchestra.run:waiting for 3000 2024-09-22T03:08:16.533 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:hints = 1 2024-09-22T03:08:16.533 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:Maintaining 16 concurrent writes of 8192 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-09-22T03:08:16.533 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:Object prefix: benchmark_data_smithi153_30331 2024-09-22T03:08:16.533 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-22T03:08:16.533 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 0 0 0 0 0 0 - 0 2024-09-22T03:08:17.533 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 1 16 376 360 2.81235 2.8125 0.0305223 0.0426999 2024-09-22T03:08:18.163 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-22T03:08:18.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:08:18.289+0000 7fec9d117640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:08:18.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:08:18.297+0000 7fec9d117640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:08:18.533 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 2 16 751 735 2.87091 2.92969 0.0581005 0.0429637 2024-09-22T03:08:19.307 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-09-22T03:08:19.308 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-22T03:08:19.308 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-09-22T03:08:19.308 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-09-22T03:08:19.438 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:08:19.428+0000 7f7757517640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:08:19.444 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:08:19.442+0000 7f7757517640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:08:19.499 INFO:teuthology.orchestra.run.smithi153.stderr:no valid command found; 10 closest matches: 2024-09-22T03:08:19.500 INFO:teuthology.orchestra.run.smithi153.stderr:config get 2024-09-22T03:08:19.500 INFO:teuthology.orchestra.run.smithi153.stderr:config help 2024-09-22T03:08:19.500 INFO:teuthology.orchestra.run.smithi153.stderr:config set ... 2024-09-22T03:08:19.500 INFO:teuthology.orchestra.run.smithi153.stderr:config show 2024-09-22T03:08:19.500 INFO:teuthology.orchestra.run.smithi153.stderr:deep_scrub 2024-09-22T03:08:19.500 INFO:teuthology.orchestra.run.smithi153.stderr:dump_historic_ops 2024-09-22T03:08:19.500 INFO:teuthology.orchestra.run.smithi153.stderr:dump_historic_slow_ops 2024-09-22T03:08:19.500 INFO:teuthology.orchestra.run.smithi153.stderr:dump_metrics [] 2024-09-22T03:08:19.500 INFO:teuthology.orchestra.run.smithi153.stderr:dump_ops_in_flight 2024-09-22T03:08:19.500 INFO:teuthology.orchestra.run.smithi153.stderr:dump_pgstate_history 2024-09-22T03:08:19.500 INFO:teuthology.orchestra.run.smithi153.stderr:Error EINVAL: invalid command 2024-09-22T03:08:19.502 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-22T03:08:19.503 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-22T03:08:19.503 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-09-22T03:08:19.533 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 3 16 1106 1090 2.83835 2.77344 0.0664991 0.0436158 2024-09-22T03:08:19.643 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:08:19.636+0000 7f05142c3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:08:19.647 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:08:19.646+0000 7f05142c3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:08:19.662 INFO:teuthology.orchestra.run.smithi153.stderr:no valid command found; 10 closest matches: 2024-09-22T03:08:19.662 INFO:teuthology.orchestra.run.smithi153.stderr:config get 2024-09-22T03:08:19.662 INFO:teuthology.orchestra.run.smithi153.stderr:config help 2024-09-22T03:08:19.663 INFO:teuthology.orchestra.run.smithi153.stderr:config set ... 2024-09-22T03:08:19.663 INFO:teuthology.orchestra.run.smithi153.stderr:config show 2024-09-22T03:08:19.663 INFO:teuthology.orchestra.run.smithi153.stderr:deep_scrub 2024-09-22T03:08:19.663 INFO:teuthology.orchestra.run.smithi153.stderr:dump_historic_ops 2024-09-22T03:08:19.663 INFO:teuthology.orchestra.run.smithi153.stderr:dump_historic_slow_ops 2024-09-22T03:08:19.663 INFO:teuthology.orchestra.run.smithi153.stderr:dump_metrics [] 2024-09-22T03:08:19.663 INFO:teuthology.orchestra.run.smithi153.stderr:dump_ops_in_flight 2024-09-22T03:08:19.663 INFO:teuthology.orchestra.run.smithi153.stderr:dump_pgstate_history 2024-09-22T03:08:19.664 INFO:teuthology.orchestra.run.smithi153.stderr:Error EINVAL: invalid command 2024-09-22T03:08:19.665 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-22T03:08:19.666 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-22T03:08:19.666 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-09-22T03:08:19.859 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:08:19.850+0000 7ff25ad4e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:08:19.865 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:08:19.863+0000 7ff25ad4e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:08:19.911 INFO:teuthology.orchestra.run.smithi153.stderr:no valid command found; 10 closest matches: 2024-09-22T03:08:19.911 INFO:teuthology.orchestra.run.smithi153.stderr:config get 2024-09-22T03:08:19.911 INFO:teuthology.orchestra.run.smithi153.stderr:config help 2024-09-22T03:08:19.911 INFO:teuthology.orchestra.run.smithi153.stderr:config set ... 2024-09-22T03:08:19.911 INFO:teuthology.orchestra.run.smithi153.stderr:config show 2024-09-22T03:08:19.911 INFO:teuthology.orchestra.run.smithi153.stderr:deep_scrub 2024-09-22T03:08:19.912 INFO:teuthology.orchestra.run.smithi153.stderr:dump_historic_ops 2024-09-22T03:08:19.912 INFO:teuthology.orchestra.run.smithi153.stderr:dump_historic_slow_ops 2024-09-22T03:08:19.912 INFO:teuthology.orchestra.run.smithi153.stderr:dump_metrics [] 2024-09-22T03:08:19.912 INFO:teuthology.orchestra.run.smithi153.stderr:dump_ops_in_flight 2024-09-22T03:08:19.912 INFO:teuthology.orchestra.run.smithi153.stderr:dump_pgstate_history 2024-09-22T03:08:19.912 INFO:teuthology.orchestra.run.smithi153.stderr:Error EINVAL: invalid command 2024-09-22T03:08:19.914 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-22T03:08:19.915 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-22T03:08:19.915 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-09-22T03:08:20.041 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:08:20.034+0000 7f137a997640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:08:20.044 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:08:20.042+0000 7f137a997640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:08:20.066 INFO:teuthology.orchestra.run.smithi153.stderr:no valid command found; 10 closest matches: 2024-09-22T03:08:20.066 INFO:teuthology.orchestra.run.smithi153.stderr:config get 2024-09-22T03:08:20.066 INFO:teuthology.orchestra.run.smithi153.stderr:config help 2024-09-22T03:08:20.066 INFO:teuthology.orchestra.run.smithi153.stderr:config set ... 2024-09-22T03:08:20.066 INFO:teuthology.orchestra.run.smithi153.stderr:config show 2024-09-22T03:08:20.066 INFO:teuthology.orchestra.run.smithi153.stderr:deep_scrub 2024-09-22T03:08:20.066 INFO:teuthology.orchestra.run.smithi153.stderr:dump_historic_ops 2024-09-22T03:08:20.066 INFO:teuthology.orchestra.run.smithi153.stderr:dump_historic_slow_ops 2024-09-22T03:08:20.066 INFO:teuthology.orchestra.run.smithi153.stderr:dump_metrics [] 2024-09-22T03:08:20.066 INFO:teuthology.orchestra.run.smithi153.stderr:dump_ops_in_flight 2024-09-22T03:08:20.067 INFO:teuthology.orchestra.run.smithi153.stderr:dump_pgstate_history 2024-09-22T03:08:20.067 INFO:teuthology.orchestra.run.smithi153.stderr:Error EINVAL: invalid command 2024-09-22T03:08:20.070 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-22T03:08:20.071 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-22T03:08:20.464 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-22T03:08:20.533 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 4 16 1486 1470 2.87089 2.96875 0.0446042 0.0432912 2024-09-22T03:08:20.607 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:08:20.604+0000 7fba3e197640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:08:20.612 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:08:20.611+0000 7fba3e197640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:08:21.533 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 5 16 1876 1860 2.90604 3.04688 0.0297927 0.0427949 2024-09-22T03:08:22.533 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 6 16 2277 2261 2.94379 3.13281 0.028251 0.0421839 2024-09-22T03:08:22.782 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-22T03:08:22.919 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:08:22.914+0000 7fa8abd57640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:08:22.926 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:08:22.924+0000 7fa8abd57640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:08:23.533 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 7 16 2652 2636 2.94174 2.92969 0.0278958 0.0423748 2024-09-22T03:08:24.533 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 8 16 3028 3012 2.94118 2.9375 0.0680943 0.0422978 2024-09-22T03:08:25.063 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-22T03:08:25.071 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-09-22T03:08:25.072 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-22T03:08:25.072 INFO:tasks.thrashosds.thrasher:Killing osd 2, live_osds are [0, 1, 2, 3] 2024-09-22T03:08:25.072 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-09-22T03:08:25.072 INFO:teuthology.orchestra.run:waiting for 300 2024-09-22T03:08:25.166 INFO:tasks.ceph.osd.2:Stopped 2024-09-22T03:08:25.197 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:08:25.194+0000 7fe20206c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:08:25.201 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:08:25.199+0000 7fe20206c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:08:25.533 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 9 16 3256 3240 2.81226 1.78125 0.0392478 0.0422864 2024-09-22T03:08:26.534 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 10 16 3256 3240 2.53104 0 - 0.0422864 2024-09-22T03:08:27.534 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 11 16 3256 3240 2.30093 0 - 0.0422864 2024-09-22T03:08:28.534 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 12 16 3256 3240 2.10918 0 - 0.0422864 2024-09-22T03:08:29.534 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 13 16 3256 3240 1.94692 0 - 0.0422864 2024-09-22T03:08:30.167 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [2] live_osds: [0, 1, 3] 2024-09-22T03:08:30.167 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-22T03:08:30.167 INFO:tasks.thrashosds.thrasher:Killing osd 3, live_osds are [0, 1, 3] 2024-09-22T03:08:30.167 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-09-22T03:08:30.167 INFO:teuthology.orchestra.run:waiting for 300 2024-09-22T03:08:30.261 INFO:tasks.ceph.osd.3:Stopped 2024-09-22T03:08:30.534 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 14 16 3256 3240 1.80785 0 - 0.0422864 2024-09-22T03:08:31.534 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 15 16 3256 3240 1.68732 0 - 0.0422864 2024-09-22T03:08:32.535 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 16 16 3256 3240 1.58186 0 - 0.0422864 2024-09-22T03:08:33.535 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 17 16 3256 3240 1.48881 0 - 0.0422864 2024-09-22T03:08:34.535 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 18 16 3256 3240 1.40609 0 - 0.0422864 2024-09-22T03:08:35.262 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [2, 3] live_osds: [0, 1] 2024-09-22T03:08:35.262 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-22T03:08:35.262 INFO:tasks.thrashosds.thrasher:Reviving osd 2 2024-09-22T03:08:35.262 INFO:tasks.ceph.osd.2:Restarting daemon 2024-09-22T03:08:35.262 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2>> /var/log/ceph/ceph-osd.2.log 2024-09-22T03:08:35.265 INFO:tasks.ceph.osd.2:Started 2024-09-22T03:08:35.265 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-22T03:08:35.374 INFO:teuthology.orchestra.run.smithi196.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-09-22T03:08:35.377 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-22T03:08:35.378 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2024-09-22T03:08:35.535 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 19 16 3256 3240 1.33209 0 - 0.0422864 2024-09-22T03:08:36.535 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:2024-09-22T03:08:36.534571+0000 min lat: 0.00806924 max lat: 0.127603 avg lat: 0.0422864 2024-09-22T03:08:36.535 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-22T03:08:36.535 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 20 16 3256 3240 1.26549 0 - 0.0422864 2024-09-22T03:08:37.535 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 21 16 3256 3240 1.20522 0 - 0.0422864 2024-09-22T03:08:38.535 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 22 16 3256 3240 1.15044 0 - 0.0422864 2024-09-22T03:08:39.535 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 23 16 3256 3240 1.10042 0 - 0.0422864 2024-09-22T03:08:40.378 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-22T03:08:40.474 INFO:teuthology.orchestra.run.smithi196.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-09-22T03:08:40.476 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-22T03:08:40.477 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2024-09-22T03:08:40.535 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 24 16 3256 3240 1.05457 0 - 0.0422864 2024-09-22T03:08:41.536 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 25 16 3256 3240 1.01238 0 - 0.0422864 2024-09-22T03:08:41.859 INFO:teuthology.orchestra.run.smithi153.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-09-22T03:08:42.536 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 26 16 3256 3240 0.973444 0 - 0.0422864 2024-09-22T03:08:43.536 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 27 16 3256 3240 0.937389 0 - 0.0422864 2024-09-22T03:08:44.536 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 28 16 3264 3248 0.906141 0.00328947 18.8132 0.0885485 2024-09-22T03:08:45.477 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-22T03:08:45.536 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 29 16 3264 3248 0.874894 0 - 0.0885485 2024-09-22T03:08:45.589 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-09-22T03:08:45.589 INFO:teuthology.orchestra.run.smithi196.stdout: "ops_in_flight": [] 2024-09-22T03:08:45.589 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-09-22T03:08:46.536 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 30 16 3264 3248 0.84573 0 - 0.0885485 2024-09-22T03:08:47.537 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 31 16 3264 3248 0.818448 0 - 0.0885485 2024-09-22T03:08:48.537 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 32 16 3264 3248 0.792871 0 - 0.0885485 2024-09-22T03:08:49.537 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 33 16 3264 3248 0.768845 0 - 0.0885485 2024-09-22T03:08:50.537 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 34 16 3264 3248 0.746232 0 - 0.0885485 2024-09-22T03:08:50.600 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [3] live_osds: [0, 1, 2] 2024-09-22T03:08:50.600 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-22T03:08:50.601 INFO:tasks.thrashosds.thrasher:Killing osd 0, live_osds are [0, 1, 2] 2024-09-22T03:08:50.601 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-09-22T03:08:50.601 INFO:teuthology.orchestra.run:waiting for 300 2024-09-22T03:08:50.698 INFO:tasks.ceph.osd.0:Stopped 2024-09-22T03:08:51.537 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 35 16 3264 3248 0.724912 0 - 0.0885485 2024-09-22T03:08:52.537 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 36 16 3264 3248 0.704775 0 - 0.0885485 2024-09-22T03:08:53.537 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 37 16 3264 3248 0.685727 0 - 0.0885485 2024-09-22T03:08:54.537 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 38 16 3264 3248 0.667681 0 - 0.0885485 2024-09-22T03:08:55.538 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 39 16 3264 3248 0.65056 0 - 0.0885485 2024-09-22T03:08:55.699 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [3, 0] live_osds: [1, 2] 2024-09-22T03:08:55.699 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-22T03:08:55.699 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-09-22T03:08:55.699 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-09-22T03:08:55.818 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:08:55.812+0000 7f174846c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:08:55.822 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:08:55.820+0000 7f174846c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:08:56.538 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:2024-09-22T03:08:56.537300+0000 min lat: 0.00806924 max lat: 18.8435 avg lat: 0.0885485 2024-09-22T03:08:56.538 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-22T03:08:56.538 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 40 16 3264 3248 0.634297 0 - 0.0885485 2024-09-22T03:08:57.538 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 41 16 3264 3248 0.618826 0 - 0.0885485 2024-09-22T03:08:58.538 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 42 16 3264 3248 0.604091 0 - 0.0885485 2024-09-22T03:08:59.538 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 43 16 3264 3248 0.590042 0 - 0.0885485 2024-09-22T03:09:00.538 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 44 16 3264 3248 0.576632 0 - 0.0885485 2024-09-22T03:09:01.538 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 45 16 3264 3248 0.563818 0 - 0.0885485 2024-09-22T03:09:02.539 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 46 16 3264 3248 0.55156 0 - 0.0885485 2024-09-22T03:09:03.539 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 47 16 3264 3248 0.539825 0 - 0.0885485 2024-09-22T03:09:04.539 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 48 16 3264 3248 0.528578 0 - 0.0885485 2024-09-22T03:09:05.539 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 49 16 3264 3248 0.517791 0 - 0.0885485 2024-09-22T03:09:06.539 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 50 16 3264 3248 0.507435 0 - 0.0885485 2024-09-22T03:09:07.539 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 51 16 3264 3248 0.497485 0 - 0.0885485 2024-09-22T03:09:08.540 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 52 16 3264 3248 0.487918 0 - 0.0885485 2024-09-22T03:09:09.539 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 53 16 3264 3248 0.478712 0 - 0.0885485 2024-09-22T03:09:10.540 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 54 16 3264 3248 0.469847 0 - 0.0885485 2024-09-22T03:09:11.540 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 55 16 3264 3248 0.461304 0 - 0.0885485 2024-09-22T03:09:12.176 INFO:teuthology.orchestra.run.smithi153.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-09-22T03:09:12.180 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-22T03:09:12.181 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-22T03:09:12.540 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 56 16 3264 3248 0.453067 0 - 0.0885485 2024-09-22T03:09:13.540 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 57 16 3264 3248 0.445118 0 - 0.0885485 2024-09-22T03:09:14.182 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-22T03:09:14.297 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:09:14.294+0000 7f487186c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:09:14.300 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:09:14.299+0000 7f487186c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:09:14.540 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 58 16 3264 3248 0.437443 0 - 0.0885485 2024-09-22T03:09:15.540 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 59 16 3264 3248 0.430029 0 - 0.0885485 2024-09-22T03:09:16.541 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:2024-09-22T03:09:16.540163+0000 min lat: 0.00806924 max lat: 18.8435 avg lat: 0.0885485 2024-09-22T03:09:16.541 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-22T03:09:16.541 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 60 16 3264 3248 0.422862 0 - 0.0885485 2024-09-22T03:09:17.541 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 61 16 3264 3248 0.415929 0 - 0.0885485 2024-09-22T03:09:18.541 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 62 16 3264 3248 0.40922 0 - 0.0885485 2024-09-22T03:09:19.541 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 63 16 3264 3248 0.402725 0 - 0.0885485 2024-09-22T03:09:19.859 INFO:teuthology.orchestra.run.smithi153.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-22T03:09:19.926 INFO:teuthology.orchestra.run.smithi153.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-09-22T03:09:19.929 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-22T03:09:19.930 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-22T03:09:20.541 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 64 16 3264 3248 0.396432 0 - 0.0885485 2024-09-22T03:09:20.829 INFO:teuthology.orchestra.run.smithi153.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-22T03:09:20.833 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-22T03:09:20.833 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-22T03:09:20.834 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-09-22T03:09:20.959 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:09:20.952+0000 7fb109b97640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:09:20.962 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:09:20.960+0000 7fb109b97640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:09:20.974 INFO:teuthology.orchestra.run.smithi153.stderr:no valid command found; 10 closest matches: 2024-09-22T03:09:20.974 INFO:teuthology.orchestra.run.smithi153.stderr:config get 2024-09-22T03:09:20.974 INFO:teuthology.orchestra.run.smithi153.stderr:config help 2024-09-22T03:09:20.974 INFO:teuthology.orchestra.run.smithi153.stderr:config set ... 2024-09-22T03:09:20.975 INFO:teuthology.orchestra.run.smithi153.stderr:config show 2024-09-22T03:09:20.975 INFO:teuthology.orchestra.run.smithi153.stderr:deep_scrub 2024-09-22T03:09:20.975 INFO:teuthology.orchestra.run.smithi153.stderr:dump_historic_ops 2024-09-22T03:09:20.975 INFO:teuthology.orchestra.run.smithi153.stderr:dump_historic_slow_ops 2024-09-22T03:09:20.975 INFO:teuthology.orchestra.run.smithi153.stderr:dump_metrics [] 2024-09-22T03:09:20.975 INFO:teuthology.orchestra.run.smithi153.stderr:dump_ops_in_flight 2024-09-22T03:09:20.975 INFO:teuthology.orchestra.run.smithi153.stderr:dump_pgstate_history 2024-09-22T03:09:20.975 INFO:teuthology.orchestra.run.smithi153.stderr:Error EINVAL: invalid command 2024-09-22T03:09:20.979 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-22T03:09:20.979 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-22T03:09:20.979 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-09-22T03:09:21.083 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:09:21.077+0000 7f55c3f17640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:09:21.087 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:09:21.085+0000 7f55c3f17640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:09:21.099 INFO:teuthology.orchestra.run.smithi153.stderr:no valid command found; 10 closest matches: 2024-09-22T03:09:21.099 INFO:teuthology.orchestra.run.smithi153.stderr:config get 2024-09-22T03:09:21.099 INFO:teuthology.orchestra.run.smithi153.stderr:config help 2024-09-22T03:09:21.099 INFO:teuthology.orchestra.run.smithi153.stderr:config set ... 2024-09-22T03:09:21.100 INFO:teuthology.orchestra.run.smithi153.stderr:config show 2024-09-22T03:09:21.100 INFO:teuthology.orchestra.run.smithi153.stderr:deep_scrub 2024-09-22T03:09:21.100 INFO:teuthology.orchestra.run.smithi153.stderr:dump_historic_ops 2024-09-22T03:09:21.100 INFO:teuthology.orchestra.run.smithi153.stderr:dump_historic_slow_ops 2024-09-22T03:09:21.100 INFO:teuthology.orchestra.run.smithi153.stderr:dump_metrics [] 2024-09-22T03:09:21.100 INFO:teuthology.orchestra.run.smithi153.stderr:dump_ops_in_flight 2024-09-22T03:09:21.100 INFO:teuthology.orchestra.run.smithi153.stderr:dump_pgstate_history 2024-09-22T03:09:21.100 INFO:teuthology.orchestra.run.smithi153.stderr:Error EINVAL: invalid command 2024-09-22T03:09:21.102 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-22T03:09:21.103 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-22T03:09:21.103 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-09-22T03:09:21.208 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:09:21.201+0000 7f44d986c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:09:21.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:09:21.210+0000 7f44d986c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:09:21.216 INFO:teuthology.orchestra.run.smithi153.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-09-22T03:09:21.219 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-22T03:09:21.219 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-22T03:09:21.541 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 65 16 3273 3257 0.391414 0.00190034 0.00848765 0.0996783 2024-09-22T03:09:21.931 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-22T03:09:22.036 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:09:22.033+0000 7fe6a5018640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:09:22.039 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:09:22.037+0000 7fe6a5018640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:09:22.043 INFO:teuthology.orchestra.run.smithi153.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-22T03:09:22.071 INFO:teuthology.orchestra.run.smithi153.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-09-22T03:09:22.074 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-22T03:09:22.074 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-22T03:09:22.541 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 66 16 3273 3257 0.385484 0 - 0.0996783 2024-09-22T03:09:23.542 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 67 16 3273 3257 0.37973 0 - 0.0996783 2024-09-22T03:09:24.075 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-22T03:09:24.192 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:09:24.189+0000 7f558846c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:09:24.195 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:09:24.193+0000 7f558846c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:09:24.199 INFO:teuthology.orchestra.run.smithi153.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-22T03:09:24.227 INFO:teuthology.orchestra.run.smithi153.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-09-22T03:09:24.230 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-22T03:09:24.230 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-22T03:09:24.542 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 68 16 3273 3257 0.374146 0 - 0.0996783 2024-09-22T03:09:25.542 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 69 16 3273 3257 0.368724 0 - 0.0996783 2024-09-22T03:09:26.220 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [3, 0] live_osds: [1, 2] 2024-09-22T03:09:26.220 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-22T03:09:26.220 INFO:tasks.thrashosds.thrasher:Reviving osd 3 2024-09-22T03:09:26.220 INFO:tasks.ceph.osd.3:Restarting daemon 2024-09-22T03:09:26.221 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2>> /var/log/ceph/ceph-osd.3.log 2024-09-22T03:09:26.223 INFO:tasks.ceph.osd.3:Started 2024-09-22T03:09:26.224 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-22T03:09:26.231 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-22T03:09:26.332 INFO:teuthology.orchestra.run.smithi196.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-09-22T03:09:26.335 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-22T03:09:26.335 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-09-22T03:09:26.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:09:26.343+0000 7fc270757640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:09:26.349 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:09:26.348+0000 7fc270757640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:09:26.353 INFO:teuthology.orchestra.run.smithi153.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-22T03:09:26.389 INFO:teuthology.orchestra.run.smithi153.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-09-22T03:09:26.392 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-22T03:09:26.392 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-22T03:09:26.542 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 70 16 3273 3257 0.363456 0 - 0.0996783 2024-09-22T03:09:27.542 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 71 16 3273 3257 0.358337 0 - 0.0996783 2024-09-22T03:09:28.393 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-22T03:09:28.508 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:09:28.505+0000 7fa3fa917640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:09:28.511 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:09:28.509+0000 7fa3fa917640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:09:28.515 INFO:teuthology.orchestra.run.smithi153.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-22T03:09:28.541 INFO:teuthology.orchestra.run.smithi153.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-09-22T03:09:28.542 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 72 16 3273 3257 0.35336 0 - 0.0996783 2024-09-22T03:09:28.545 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-22T03:09:28.545 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-22T03:09:29.542 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 73 16 3273 3257 0.34852 0 - 0.0996783 2024-09-22T03:09:30.543 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 74 16 3273 3257 0.34381 0 - 0.0996783 2024-09-22T03:09:30.546 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-22T03:09:30.663 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:09:30.660+0000 7f0361a2c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:09:30.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:09:30.664+0000 7f0361a2c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:09:30.670 INFO:teuthology.orchestra.run.smithi153.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-22T03:09:30.698 INFO:teuthology.orchestra.run.smithi153.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-09-22T03:09:30.701 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-22T03:09:30.702 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-22T03:09:31.337 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-22T03:09:31.434 INFO:teuthology.orchestra.run.smithi196.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-09-22T03:09:31.437 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-22T03:09:31.437 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-09-22T03:09:31.543 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 75 16 3273 3257 0.339226 0 - 0.0996783 2024-09-22T03:09:32.543 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 76 16 3273 3257 0.334762 0 - 0.0996783 2024-09-22T03:09:32.703 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-22T03:09:32.819 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:09:32.816+0000 7f31c6d97640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:09:32.822 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:09:32.821+0000 7f31c6d97640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:09:32.826 INFO:teuthology.orchestra.run.smithi153.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-22T03:09:32.852 INFO:teuthology.orchestra.run.smithi153.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-09-22T03:09:32.855 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-22T03:09:32.856 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-22T03:09:33.543 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 77 16 3273 3257 0.330415 0 - 0.0996783 2024-09-22T03:09:34.543 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 78 16 3273 3257 0.326179 0 - 0.0996783 2024-09-22T03:09:34.857 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-22T03:09:34.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:09:34.969+0000 7f8751f97640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:09:34.975 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:09:34.974+0000 7f8751f97640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:09:34.979 INFO:teuthology.orchestra.run.smithi153.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-22T03:09:35.006 INFO:teuthology.orchestra.run.smithi153.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-09-22T03:09:35.010 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-22T03:09:35.010 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-22T03:09:35.543 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 79 16 3273 3257 0.32205 0 - 0.0996783 2024-09-22T03:09:36.438 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-22T03:09:36.543 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:2024-09-22T03:09:36.543024+0000 min lat: 0.00806924 max lat: 36.9595 avg lat: 0.0996783 2024-09-22T03:09:36.543 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-22T03:09:36.544 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 80 16 3273 3257 0.318024 0 - 0.0996783 2024-09-22T03:09:36.546 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-09-22T03:09:36.546 INFO:teuthology.orchestra.run.smithi196.stdout: "ops_in_flight": [ 2024-09-22T03:09:36.546 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-22T03:09:36.546 INFO:teuthology.orchestra.run.smithi196.stdout: "type": "client_request", 2024-09-22T03:09:36.546 INFO:teuthology.orchestra.run.smithi196.stdout: "id": 23, 2024-09-22T03:09:36.546 INFO:teuthology.orchestra.run.smithi196.stdout: "detail": { 2024-09-22T03:09:36.546 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.546 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:09:36.546 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:09:35.896330+0000" 2024-09-22T03:09:36.547 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.547 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.547 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:09:36.547 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:09:35.896337+0000" 2024-09-22T03:09:36.547 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.547 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.547 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.547 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.547 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.547 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.547 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:09:36.547 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:09:36.005043+0000" 2024-09-22T03:09:36.547 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.548 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.548 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.548 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.548 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.548 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.548 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:09:36.548 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.548 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.548 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.548 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.548 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.548 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.548 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.549 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:09:36.549 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:09:36.005778+0000" 2024-09-22T03:09:36.549 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.549 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.549 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.549 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.549 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.549 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.549 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:09:36.549 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:09:36.006394+0000" 2024-09-22T03:09:36.549 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.549 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.549 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.550 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.550 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.550 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.550 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:09:36.550 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.550 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.550 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.550 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.550 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.550 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.550 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.550 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:09:36.550 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.550 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.551 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.551 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:09:36.551 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:09:36.006933+0000" 2024-09-22T03:09:36.551 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.551 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.551 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.551 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.551 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.551 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.551 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:09:36.551 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.551 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.551 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.551 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.552 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.552 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.552 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.552 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:09:36.552 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:09:36.007132+0000" 2024-09-22T03:09:36.552 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.552 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.552 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.552 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.552 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.552 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.552 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:09:36.552 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:09:36.007929+0000", 2024-09-22T03:09:36.553 INFO:teuthology.orchestra.run.smithi196.stdout: "blocker": { 2024-09-22T03:09:36.553 INFO:teuthology.orchestra.run.smithi196.stdout: "op_type": "PGActivationBlocker", 2024-09-22T03:09:36.553 INFO:teuthology.orchestra.run.smithi196.stdout: "detail": { 2024-09-22T03:09:36.553 INFO:teuthology.orchestra.run.smithi196.stdout: "pgid": "3.7" 2024-09-22T03:09:36.553 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:09:36.553 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:09:36.553 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.553 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.553 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.553 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.553 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.553 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.553 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:09:36.553 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.554 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.554 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.554 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.554 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.554 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.554 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.554 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:09:36.554 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.554 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.554 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.554 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.554 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.554 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.555 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.555 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:09:36.555 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.555 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.555 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.555 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.555 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.555 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.555 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.555 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:09:36.555 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.555 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.555 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.555 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.556 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.556 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.556 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.556 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:09:36.556 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.556 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.556 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.556 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.556 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.556 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.556 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.556 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:09:36.556 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.557 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.557 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.557 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.557 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.557 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.557 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.557 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:09:36.557 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.558 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.558 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.558 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.558 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.558 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.558 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.558 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:09:36.558 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.558 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.558 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.558 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.558 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.559 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.559 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.559 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:09:36.559 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.559 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.559 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.559 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.559 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.559 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.559 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.559 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:09:36.559 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.559 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:09:36.559 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:09:36.560 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.560 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-22T03:09:36.560 INFO:teuthology.orchestra.run.smithi196.stdout: "type": "client_request", 2024-09-22T03:09:36.560 INFO:teuthology.orchestra.run.smithi196.stdout: "id": 24, 2024-09-22T03:09:36.560 INFO:teuthology.orchestra.run.smithi196.stdout: "detail": { 2024-09-22T03:09:36.560 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.560 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:09:36.560 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:09:35.897237+0000" 2024-09-22T03:09:36.560 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.560 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.560 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:09:36.560 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:09:35.897244+0000" 2024-09-22T03:09:36.560 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.560 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.561 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.561 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.561 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.561 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.561 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:09:36.561 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:09:36.006291+0000" 2024-09-22T03:09:36.561 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.561 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.561 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.561 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.561 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.561 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.561 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:09:36.561 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.562 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.562 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.562 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.562 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.562 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.562 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.562 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:09:36.562 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:09:36.006459+0000" 2024-09-22T03:09:36.562 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.562 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.562 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.563 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.563 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.563 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.563 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:09:36.563 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:09:36.007636+0000" 2024-09-22T03:09:36.563 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.563 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.563 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.563 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.563 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.563 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.563 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:09:36.564 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.564 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.564 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.564 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.564 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.564 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.565 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.565 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:09:36.565 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.565 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.565 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.565 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:09:36.565 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:09:36.007871+0000" 2024-09-22T03:09:36.566 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.566 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.566 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.566 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.566 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.566 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.566 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:09:36.567 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.567 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.567 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.567 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.567 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.568 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.568 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.568 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:09:36.568 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:09:36.009663+0000", 2024-09-22T03:09:36.568 INFO:teuthology.orchestra.run.smithi196.stdout: "blocker": { 2024-09-22T03:09:36.568 INFO:teuthology.orchestra.run.smithi196.stdout: "op_type": "CommonPGPipeline:::wait_for_active", 2024-09-22T03:09:36.568 INFO:teuthology.orchestra.run.smithi196.stdout: "detail": { 2024-09-22T03:09:36.569 INFO:teuthology.orchestra.run.smithi196.stdout: "waiting": 1, 2024-09-22T03:09:36.569 INFO:teuthology.orchestra.run.smithi196.stdout: "held_by_operation_id": 23 2024-09-22T03:09:36.569 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:09:36.569 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:09:36.569 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.569 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.570 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.570 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.570 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.570 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.570 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:09:36.570 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.571 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.571 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.571 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.571 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.571 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.571 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.571 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:09:36.571 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.571 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.571 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.572 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.572 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.572 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.572 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.572 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:09:36.572 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.572 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.572 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.572 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.572 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.572 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.572 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.572 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:09:36.573 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.573 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.573 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.573 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.573 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.573 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.573 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.573 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:09:36.573 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.573 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.573 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.573 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.573 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.573 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.574 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.574 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:09:36.574 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.574 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.574 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.574 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.574 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.574 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.574 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.574 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:09:36.574 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.574 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.574 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.574 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.575 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.575 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.575 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.575 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:09:36.575 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.575 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.575 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.575 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.575 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.575 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.575 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.575 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:09:36.575 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.576 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.576 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.576 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.576 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.576 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.576 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.576 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:09:36.576 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.576 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.576 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.576 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.576 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.576 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.576 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.576 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:09:36.576 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.577 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:09:36.577 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:09:36.577 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.577 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-22T03:09:36.577 INFO:teuthology.orchestra.run.smithi196.stdout: "type": "client_request", 2024-09-22T03:09:36.577 INFO:teuthology.orchestra.run.smithi196.stdout: "id": 25, 2024-09-22T03:09:36.577 INFO:teuthology.orchestra.run.smithi196.stdout: "detail": { 2024-09-22T03:09:36.577 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.577 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:09:36.577 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:09:35.897470+0000" 2024-09-22T03:09:36.577 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.577 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.577 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:09:36.577 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:09:35.897477+0000" 2024-09-22T03:09:36.577 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.578 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.578 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.578 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.578 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.578 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.578 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:09:36.578 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:09:36.006648+0000" 2024-09-22T03:09:36.578 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.578 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.578 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.578 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.578 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.578 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.578 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:09:36.579 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.579 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.579 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.579 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.579 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.579 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.579 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.579 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:09:36.579 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:09:36.007960+0000" 2024-09-22T03:09:36.579 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.579 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.579 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.579 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.579 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.579 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.580 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:09:36.580 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:09:36.009354+0000" 2024-09-22T03:09:36.580 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.580 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.580 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.580 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.580 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.580 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.580 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:09:36.580 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.580 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.580 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.580 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.580 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.580 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.581 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.581 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:09:36.581 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.581 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.581 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.581 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:09:36.581 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:09:36.010627+0000", 2024-09-22T03:09:36.581 INFO:teuthology.orchestra.run.smithi196.stdout: "blocker": { 2024-09-22T03:09:36.581 INFO:teuthology.orchestra.run.smithi196.stdout: "op_type": "ClientRequest::PGPipeline::await_map", 2024-09-22T03:09:36.581 INFO:teuthology.orchestra.run.smithi196.stdout: "detail": { 2024-09-22T03:09:36.581 INFO:teuthology.orchestra.run.smithi196.stdout: "waiting": 1, 2024-09-22T03:09:36.581 INFO:teuthology.orchestra.run.smithi196.stdout: "held_by_operation_id": 24 2024-09-22T03:09:36.581 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:09:36.581 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:09:36.581 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.582 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.582 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.582 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.582 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.582 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.582 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:09:36.582 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.582 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.582 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.582 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.582 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.582 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.582 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.583 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:09:36.583 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.583 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.583 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.583 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.583 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.583 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.583 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.583 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:09:36.583 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.583 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.583 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.583 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.583 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.584 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.584 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.584 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:09:36.584 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.584 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.584 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.584 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.584 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.584 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.584 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.584 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:09:36.584 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.584 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.584 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.585 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.585 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.585 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.585 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.585 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:09:36.585 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.585 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.585 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.585 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.585 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.585 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.585 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.585 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:09:36.586 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.586 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.586 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.586 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.586 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.586 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.586 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.586 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:09:36.586 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.586 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.586 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.586 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.586 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.587 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.587 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.587 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:09:36.587 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.587 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.587 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.587 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.587 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.587 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.587 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.587 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:09:36.587 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.587 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.587 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.588 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.588 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.588 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.588 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.588 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:09:36.588 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.588 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.588 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.588 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.588 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.588 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.588 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.589 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:09:36.589 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.589 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.589 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.589 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.589 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.589 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.589 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.589 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:09:36.589 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.589 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:09:36.589 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:09:36.589 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.590 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-22T03:09:36.590 INFO:teuthology.orchestra.run.smithi196.stdout: "type": "client_request", 2024-09-22T03:09:36.590 INFO:teuthology.orchestra.run.smithi196.stdout: "id": 26, 2024-09-22T03:09:36.590 INFO:teuthology.orchestra.run.smithi196.stdout: "detail": { 2024-09-22T03:09:36.590 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.590 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:09:36.590 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:09:35.898723+0000" 2024-09-22T03:09:36.590 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.590 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.590 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:09:36.590 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:09:35.898731+0000" 2024-09-22T03:09:36.591 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.591 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.591 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.591 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.591 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.591 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.591 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:09:36.591 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:09:36.008083+0000" 2024-09-22T03:09:36.592 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.592 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.592 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.592 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.592 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.592 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.593 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:09:36.593 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.593 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.593 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.593 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.593 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.593 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.594 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.594 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:09:36.594 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:09:36.009585+0000" 2024-09-22T03:09:36.594 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.594 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.594 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.594 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.594 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.595 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.595 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:09:36.595 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:09:36.010845+0000", 2024-09-22T03:09:36.595 INFO:teuthology.orchestra.run.smithi196.stdout: "blocker": { 2024-09-22T03:09:36.595 INFO:teuthology.orchestra.run.smithi196.stdout: "op_type": "PerShardPipeline::create_or_wait_pg", 2024-09-22T03:09:36.595 INFO:teuthology.orchestra.run.smithi196.stdout: "detail": { 2024-09-22T03:09:36.595 INFO:teuthology.orchestra.run.smithi196.stdout: "waiting": 1, 2024-09-22T03:09:36.596 INFO:teuthology.orchestra.run.smithi196.stdout: "held_by_operation_id": 25 2024-09-22T03:09:36.596 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:09:36.596 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:09:36.596 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.596 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.596 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.597 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.597 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.597 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.598 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:09:36.598 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.598 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.598 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.598 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.598 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.598 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.598 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.599 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:09:36.599 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.599 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.599 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.599 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:09:36.599 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.599 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.599 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.599 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.599 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.599 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.599 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.599 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:09:36.600 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.600 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.600 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.600 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.600 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.600 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.600 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.600 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:09:36.600 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.600 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.600 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.600 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.601 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.601 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.601 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.601 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:09:36.601 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.601 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.601 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.601 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.601 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.601 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.601 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.601 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:09:36.601 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.601 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.602 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.602 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.602 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.602 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.602 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.602 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:09:36.602 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.602 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.602 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.602 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.602 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.602 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.603 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.603 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:09:36.603 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.603 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.603 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.603 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.603 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.603 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.603 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.603 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:09:36.603 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.603 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.603 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.604 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.604 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.604 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.604 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.604 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:09:36.604 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.604 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.604 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.604 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.604 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.604 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.604 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.604 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:09:36.605 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.605 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.605 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.605 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.605 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.605 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.605 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.605 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:09:36.605 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.605 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.605 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.605 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.605 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.605 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.606 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.606 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:09:36.606 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.606 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.606 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.606 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.606 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.606 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.606 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.606 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:09:36.606 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.606 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.606 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.607 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.607 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.607 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.607 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.607 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:09:36.607 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.607 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:09:36.607 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:09:36.607 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.607 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-22T03:09:36.607 INFO:teuthology.orchestra.run.smithi196.stdout: "type": "client_request", 2024-09-22T03:09:36.607 INFO:teuthology.orchestra.run.smithi196.stdout: "id": 27, 2024-09-22T03:09:36.607 INFO:teuthology.orchestra.run.smithi196.stdout: "detail": { 2024-09-22T03:09:36.607 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.608 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:09:36.608 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:09:35.900188+0000" 2024-09-22T03:09:36.608 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.608 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.608 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:09:36.608 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:09:35.900195+0000" 2024-09-22T03:09:36.608 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.608 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.608 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.608 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.608 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.608 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.608 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:09:36.608 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:09:36.009605+0000" 2024-09-22T03:09:36.609 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.609 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.609 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.609 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.609 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.609 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.609 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:09:36.609 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.609 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.609 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.609 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.609 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.609 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.610 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.610 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:09:36.610 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:09:36.011121+0000", 2024-09-22T03:09:36.610 INFO:teuthology.orchestra.run.smithi196.stdout: "blocker": { 2024-09-22T03:09:36.610 INFO:teuthology.orchestra.run.smithi196.stdout: "op_type": "ConnectionPipeline::get_pg_mapping", 2024-09-22T03:09:36.610 INFO:teuthology.orchestra.run.smithi196.stdout: "detail": { 2024-09-22T03:09:36.610 INFO:teuthology.orchestra.run.smithi196.stdout: "waiting": 1, 2024-09-22T03:09:36.610 INFO:teuthology.orchestra.run.smithi196.stdout: "held_by_operation_id": 26 2024-09-22T03:09:36.610 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:09:36.610 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:09:36.610 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.610 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.610 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.611 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.611 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.611 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.611 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:09:36.611 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.611 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.611 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.611 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.611 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.611 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.611 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.611 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:09:36.611 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.611 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.612 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.612 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.612 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.612 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.612 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.612 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:09:36.612 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.612 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.612 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.612 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:09:36.612 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.612 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.612 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.612 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.612 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.613 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.613 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.613 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:09:36.613 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.613 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.613 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.613 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.613 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.613 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.613 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.613 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:09:36.613 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.613 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.613 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.614 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.614 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.614 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.614 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.614 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:09:36.614 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.614 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.614 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.614 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.614 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.614 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.614 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.615 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:09:36.615 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.615 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.615 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.615 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.615 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.615 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.615 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.615 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:09:36.615 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.615 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.615 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.615 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.616 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.616 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.616 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.616 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:09:36.616 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.616 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.616 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.616 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.616 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.616 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.616 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.616 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:09:36.616 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.616 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.617 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.617 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.617 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.617 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.617 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.617 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:09:36.617 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.617 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.617 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.617 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.617 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.617 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.617 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.618 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:09:36.618 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.618 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.618 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.618 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.618 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.618 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.618 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.618 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:09:36.618 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.618 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.618 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.618 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.618 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.619 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.619 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.619 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:09:36.619 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.619 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.619 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.619 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.619 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.619 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.619 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.619 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:09:36.619 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.619 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.620 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.620 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.620 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.620 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.620 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.620 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:09:36.620 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.620 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:09:36.620 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:09:36.620 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.620 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-22T03:09:36.620 INFO:teuthology.orchestra.run.smithi196.stdout: "type": "client_request", 2024-09-22T03:09:36.620 INFO:teuthology.orchestra.run.smithi196.stdout: "id": 28, 2024-09-22T03:09:36.621 INFO:teuthology.orchestra.run.smithi196.stdout: "detail": { 2024-09-22T03:09:36.621 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.621 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:09:36.621 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:09:35.901653+0000" 2024-09-22T03:09:36.621 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.621 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.621 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:09:36.621 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:09:35.901660+0000" 2024-09-22T03:09:36.621 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.621 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.621 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.621 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.621 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.621 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.621 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:09:36.622 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:09:36.011253+0000", 2024-09-22T03:09:36.622 INFO:teuthology.orchestra.run.smithi196.stdout: "blocker": { 2024-09-22T03:09:36.622 INFO:teuthology.orchestra.run.smithi196.stdout: "op_type": "ConnectionPipeline::await_map", 2024-09-22T03:09:36.622 INFO:teuthology.orchestra.run.smithi196.stdout: "detail": { 2024-09-22T03:09:36.622 INFO:teuthology.orchestra.run.smithi196.stdout: "waiting": 1, 2024-09-22T03:09:36.622 INFO:teuthology.orchestra.run.smithi196.stdout: "held_by_operation_id": 27 2024-09-22T03:09:36.622 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:09:36.622 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:09:36.622 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.622 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.622 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.622 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.622 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.623 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.623 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:09:36.623 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.623 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.623 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.623 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.623 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.623 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.623 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.623 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:09:36.623 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.623 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.623 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.623 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.624 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.624 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.624 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.624 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:09:36.624 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.624 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.624 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.624 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.624 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.624 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.624 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.624 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:09:36.624 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.624 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.625 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.625 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.625 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.625 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.625 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.625 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:09:36.625 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.625 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.625 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.625 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:09:36.625 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.625 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.625 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.625 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.625 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.626 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.626 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.626 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:09:36.626 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.626 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.626 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.626 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.626 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.626 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.626 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.626 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:09:36.626 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.626 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.627 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.627 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.627 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.627 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.627 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.627 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:09:36.627 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.627 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.627 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.627 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.627 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.627 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.627 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.628 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:09:36.628 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.628 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.628 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.628 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.628 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.628 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.628 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.628 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:09:36.628 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.628 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.628 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.628 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.628 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.629 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.629 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.629 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:09:36.629 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.629 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.629 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.629 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.629 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.629 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.629 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.629 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:09:36.629 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.629 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.629 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.630 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.630 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.630 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.630 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.630 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:09:36.630 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.630 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.630 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.630 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.630 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.630 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.630 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.631 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:09:36.631 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.631 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.631 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.631 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.631 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.631 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.631 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.631 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:09:36.632 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.636 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.636 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.636 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.636 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.636 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.636 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.637 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:09:36.637 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.637 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.637 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.637 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.637 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.637 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.637 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.637 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:09:36.637 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.637 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.637 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.637 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.637 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.638 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.638 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.638 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:09:36.638 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.638 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:09:36.638 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:09:36.638 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.638 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-22T03:09:36.638 INFO:teuthology.orchestra.run.smithi196.stdout: "type": "client_request", 2024-09-22T03:09:36.638 INFO:teuthology.orchestra.run.smithi196.stdout: "id": 29, 2024-09-22T03:09:36.638 INFO:teuthology.orchestra.run.smithi196.stdout: "detail": { 2024-09-22T03:09:36.638 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.638 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:09:36.639 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:09:35.901896+0000" 2024-09-22T03:09:36.639 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.639 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.639 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:09:36.639 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:09:35.901903+0000", 2024-09-22T03:09:36.639 INFO:teuthology.orchestra.run.smithi196.stdout: "blocker": { 2024-09-22T03:09:36.639 INFO:teuthology.orchestra.run.smithi196.stdout: "op_type": "ConnectionPipeline::await_active", 2024-09-22T03:09:36.639 INFO:teuthology.orchestra.run.smithi196.stdout: "detail": { 2024-09-22T03:09:36.639 INFO:teuthology.orchestra.run.smithi196.stdout: "waiting": 1, 2024-09-22T03:09:36.639 INFO:teuthology.orchestra.run.smithi196.stdout: "held_by_operation_id": 28 2024-09-22T03:09:36.639 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:09:36.639 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:09:36.639 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.639 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.640 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.640 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.640 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.640 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.640 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:09:36.640 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.640 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.640 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.640 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.640 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.640 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.640 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.640 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:09:36.641 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.641 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.641 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.641 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.641 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.641 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.641 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.641 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:09:36.641 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.641 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.641 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.641 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.641 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.641 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.642 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.642 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:09:36.642 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.642 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.642 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.642 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.642 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.642 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.642 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.642 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:09:36.642 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.642 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.642 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.643 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.643 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.643 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.643 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.643 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:09:36.643 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.643 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.643 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.643 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:09:36.643 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.643 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.643 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.643 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.644 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.644 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.644 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.644 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:09:36.644 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.644 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.644 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.644 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.644 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.644 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.644 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.644 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:09:36.644 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.644 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.645 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.645 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.645 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.645 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.645 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.645 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:09:36.645 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.645 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.645 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.645 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.645 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.645 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.645 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.645 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:09:36.646 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.646 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.646 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.646 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.646 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.646 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.646 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.646 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:09:36.646 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.646 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.646 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.646 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.646 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.647 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.647 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.647 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:09:36.647 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.647 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.647 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.647 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.647 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.647 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.647 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.647 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:09:36.647 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.648 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.648 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.648 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.648 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.648 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.648 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.648 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:09:36.648 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.648 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.648 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.648 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.648 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.649 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.649 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.649 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:09:36.649 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.649 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.649 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.649 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.649 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.649 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.649 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.650 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:09:36.650 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.650 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.650 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.650 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.650 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.650 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.650 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.651 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:09:36.651 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.651 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.651 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.651 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.651 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.651 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.651 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.651 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:09:36.651 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.651 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.651 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.651 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.652 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.652 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.652 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.652 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:09:36.652 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.652 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:09:36.652 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:09:36.652 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.652 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-22T03:09:36.652 INFO:teuthology.orchestra.run.smithi196.stdout: "type": "client_request", 2024-09-22T03:09:36.652 INFO:teuthology.orchestra.run.smithi196.stdout: "id": 15, 2024-09-22T03:09:36.652 INFO:teuthology.orchestra.run.smithi196.stdout: "detail": { 2024-09-22T03:09:36.653 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.653 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:09:36.653 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:09:35.889450+0000" 2024-09-22T03:09:36.653 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.653 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.653 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:09:36.653 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:09:35.889489+0000" 2024-09-22T03:09:36.653 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.653 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.653 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.653 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.653 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.653 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.653 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:09:36.654 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:09:35.906045+0000" 2024-09-22T03:09:36.654 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.654 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.654 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.654 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.654 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.654 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.654 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:09:36.654 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:09:35.906240+0000" 2024-09-22T03:09:36.654 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.654 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.654 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.654 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.655 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.655 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.655 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:09:36.655 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:09:36.000934+0000" 2024-09-22T03:09:36.655 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.655 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.655 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.655 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.655 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.655 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.655 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:09:36.655 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:09:36.001704+0000" 2024-09-22T03:09:36.655 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.655 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.655 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.656 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.656 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.656 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.656 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:09:36.656 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.656 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.656 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.656 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.656 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.656 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.656 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.656 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:09:36.656 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.657 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.657 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.657 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:09:36.657 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:09:36.002128+0000" 2024-09-22T03:09:36.657 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.657 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.657 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.657 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.657 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.657 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.657 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:09:36.657 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.657 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.658 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.658 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.658 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.658 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.658 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.658 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:09:36.658 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:09:36.002359+0000" 2024-09-22T03:09:36.658 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.658 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.658 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.658 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.658 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.658 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.659 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:09:36.659 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:09:36.002552+0000", 2024-09-22T03:09:36.659 INFO:teuthology.orchestra.run.smithi196.stdout: "blocker": { 2024-09-22T03:09:36.659 INFO:teuthology.orchestra.run.smithi196.stdout: "op_type": "PGActivationBlocker", 2024-09-22T03:09:36.659 INFO:teuthology.orchestra.run.smithi196.stdout: "detail": { 2024-09-22T03:09:36.659 INFO:teuthology.orchestra.run.smithi196.stdout: "pgid": "3.b" 2024-09-22T03:09:36.659 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:09:36.659 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:09:36.659 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.659 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.659 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.659 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.660 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.660 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.660 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:09:36.660 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.660 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.660 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.660 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.660 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.660 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.660 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.660 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:09:36.660 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.660 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.661 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.661 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.661 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.661 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.661 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.661 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:09:36.661 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.661 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.661 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.661 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.661 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.661 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.661 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.662 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:09:36.662 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.662 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.662 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.662 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.662 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.662 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.662 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.662 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:09:36.662 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.662 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.663 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.663 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.663 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.663 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.663 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.663 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:09:36.663 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.663 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.663 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.663 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.663 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.663 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.663 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.664 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:09:36.664 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.664 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.664 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.664 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.664 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.664 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.664 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.664 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:09:36.664 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.664 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.664 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.664 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.665 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.665 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.665 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.665 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:09:36.665 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.665 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.665 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.665 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.665 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.665 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.666 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.666 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:09:36.666 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.666 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:09:36.666 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:09:36.666 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.666 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-22T03:09:36.667 INFO:teuthology.orchestra.run.smithi196.stdout: "type": "client_request", 2024-09-22T03:09:36.667 INFO:teuthology.orchestra.run.smithi196.stdout: "id": 16, 2024-09-22T03:09:36.667 INFO:teuthology.orchestra.run.smithi196.stdout: "detail": { 2024-09-22T03:09:36.667 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.667 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:09:36.667 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:09:35.889910+0000" 2024-09-22T03:09:36.668 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.668 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.668 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:09:36.668 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:09:35.889922+0000" 2024-09-22T03:09:36.668 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.668 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.669 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.669 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.669 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.669 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.669 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:09:36.669 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:09:35.906337+0000" 2024-09-22T03:09:36.669 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.670 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.670 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.670 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.670 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.670 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.670 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:09:36.671 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.671 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.671 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.671 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.671 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.671 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.671 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.672 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:09:36.672 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:09:36.001334+0000" 2024-09-22T03:09:36.672 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.672 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.672 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.672 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.672 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.672 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.672 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:09:36.672 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:09:36.002657+0000" 2024-09-22T03:09:36.672 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.672 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.673 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.673 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.673 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.673 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.673 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:09:36.673 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.673 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.673 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.673 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.673 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.673 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.673 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.673 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:09:36.674 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.674 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.674 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.674 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:09:36.674 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:09:36.003000+0000" 2024-09-22T03:09:36.674 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.674 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.674 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.674 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.674 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.674 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.674 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:09:36.674 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.674 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.675 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.675 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.675 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.675 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.675 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.675 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:09:36.675 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:09:36.003211+0000", 2024-09-22T03:09:36.675 INFO:teuthology.orchestra.run.smithi196.stdout: "blocker": { 2024-09-22T03:09:36.675 INFO:teuthology.orchestra.run.smithi196.stdout: "op_type": "CommonPGPipeline:::wait_for_active", 2024-09-22T03:09:36.675 INFO:teuthology.orchestra.run.smithi196.stdout: "detail": { 2024-09-22T03:09:36.675 INFO:teuthology.orchestra.run.smithi196.stdout: "waiting": 1, 2024-09-22T03:09:36.675 INFO:teuthology.orchestra.run.smithi196.stdout: "held_by_operation_id": 15 2024-09-22T03:09:36.675 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:09:36.676 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:09:36.676 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.676 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.676 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.676 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.676 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.676 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.676 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:09:36.676 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.676 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.676 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.676 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.676 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.676 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.676 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.677 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:09:36.677 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.677 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.677 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.677 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.677 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.677 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.677 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.677 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:09:36.677 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.677 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.677 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.677 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.677 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.678 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.678 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.678 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:09:36.678 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.678 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.678 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.678 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.678 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.678 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.678 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.678 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:09:36.678 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.678 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.679 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.679 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.679 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.679 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.679 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.679 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:09:36.679 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.679 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.679 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.679 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.679 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.679 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.679 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.679 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:09:36.680 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.680 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.680 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.680 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.680 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.680 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.680 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.680 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:09:36.680 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.680 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.680 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.680 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.680 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.680 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.681 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.681 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:09:36.681 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.681 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.681 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.681 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.681 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.681 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.681 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.681 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:09:36.681 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.681 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.681 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.681 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.682 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.682 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.682 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.682 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:09:36.682 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.682 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:09:36.682 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:09:36.682 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.682 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-22T03:09:36.682 INFO:teuthology.orchestra.run.smithi196.stdout: "type": "client_request", 2024-09-22T03:09:36.682 INFO:teuthology.orchestra.run.smithi196.stdout: "id": 17, 2024-09-22T03:09:36.682 INFO:teuthology.orchestra.run.smithi196.stdout: "detail": { 2024-09-22T03:09:36.682 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.683 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:09:36.683 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:09:35.891960+0000" 2024-09-22T03:09:36.683 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.683 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.683 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:09:36.683 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:09:35.891967+0000" 2024-09-22T03:09:36.683 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.683 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.683 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.683 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.683 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.683 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.683 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:09:36.683 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:09:36.001587+0000" 2024-09-22T03:09:36.684 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.684 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.684 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.684 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.684 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.684 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.684 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:09:36.684 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.684 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.684 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.684 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.684 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.684 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.684 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.685 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:09:36.685 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:09:36.001933+0000" 2024-09-22T03:09:36.685 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.685 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.685 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.685 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.685 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.685 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.685 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:09:36.685 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:09:36.003278+0000" 2024-09-22T03:09:36.685 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.685 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.685 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.686 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.686 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.686 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.686 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:09:36.686 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.686 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.686 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.686 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.686 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.686 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.686 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.686 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:09:36.686 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.686 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.687 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.687 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:09:36.687 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:09:36.003604+0000", 2024-09-22T03:09:36.687 INFO:teuthology.orchestra.run.smithi196.stdout: "blocker": { 2024-09-22T03:09:36.687 INFO:teuthology.orchestra.run.smithi196.stdout: "op_type": "ClientRequest::PGPipeline::await_map", 2024-09-22T03:09:36.687 INFO:teuthology.orchestra.run.smithi196.stdout: "detail": { 2024-09-22T03:09:36.687 INFO:teuthology.orchestra.run.smithi196.stdout: "waiting": 1, 2024-09-22T03:09:36.687 INFO:teuthology.orchestra.run.smithi196.stdout: "held_by_operation_id": 16 2024-09-22T03:09:36.687 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:09:36.687 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:09:36.687 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.687 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.687 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.687 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.688 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.688 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.688 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:09:36.688 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.688 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.688 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.688 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.688 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.688 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.688 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.688 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:09:36.688 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.688 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.689 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.689 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.689 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.689 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.689 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.689 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:09:36.689 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.689 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.689 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.689 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.689 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.689 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.689 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.689 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:09:36.690 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.690 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.690 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.690 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.690 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.690 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.690 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.690 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:09:36.690 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.690 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.690 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.690 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.690 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.691 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.691 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.691 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:09:36.691 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.691 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.691 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.691 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.691 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.691 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.691 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.691 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:09:36.691 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.691 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.692 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.692 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.692 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.692 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.692 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.692 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:09:36.692 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.692 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.692 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.692 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.692 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.692 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.692 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.692 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:09:36.693 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.693 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.693 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.693 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.693 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.693 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.693 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.693 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:09:36.693 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.693 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.693 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.693 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.693 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.693 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.694 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.694 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:09:36.694 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.694 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.694 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.694 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.694 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.694 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.694 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.694 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:09:36.694 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.694 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.694 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.695 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.695 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.695 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.695 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.695 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:09:36.695 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.695 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:09:36.695 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:09:36.695 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.695 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-22T03:09:36.695 INFO:teuthology.orchestra.run.smithi196.stdout: "type": "client_request", 2024-09-22T03:09:36.695 INFO:teuthology.orchestra.run.smithi196.stdout: "id": 18, 2024-09-22T03:09:36.695 INFO:teuthology.orchestra.run.smithi196.stdout: "detail": { 2024-09-22T03:09:36.695 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.696 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:09:36.696 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:09:35.894748+0000" 2024-09-22T03:09:36.696 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.696 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.696 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:09:36.696 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:09:35.894754+0000" 2024-09-22T03:09:36.696 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.696 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.696 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.696 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.696 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.696 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.696 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:09:36.696 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:09:36.002660+0000" 2024-09-22T03:09:36.697 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.697 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.697 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.697 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.697 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.697 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.697 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:09:36.697 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.697 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.697 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.697 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.697 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.697 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.697 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.698 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:09:36.698 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:09:36.003344+0000" 2024-09-22T03:09:36.698 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.698 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.698 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.698 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.698 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.698 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.698 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:09:36.698 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:09:36.004610+0000", 2024-09-22T03:09:36.698 INFO:teuthology.orchestra.run.smithi196.stdout: "blocker": { 2024-09-22T03:09:36.698 INFO:teuthology.orchestra.run.smithi196.stdout: "op_type": "PerShardPipeline::create_or_wait_pg", 2024-09-22T03:09:36.698 INFO:teuthology.orchestra.run.smithi196.stdout: "detail": { 2024-09-22T03:09:36.699 INFO:teuthology.orchestra.run.smithi196.stdout: "waiting": 5, 2024-09-22T03:09:36.699 INFO:teuthology.orchestra.run.smithi196.stdout: "held_by_operation_id": 17 2024-09-22T03:09:36.699 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:09:36.699 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:09:36.699 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.699 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.699 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.699 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.699 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.699 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.699 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:09:36.699 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.699 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.699 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.700 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.700 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.700 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.700 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.700 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:09:36.700 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.700 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.700 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.700 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:09:36.700 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.700 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.700 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.700 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.701 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.701 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.701 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.701 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:09:36.701 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.701 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.701 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.701 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.701 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.701 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.701 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.701 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:09:36.701 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.701 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.702 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.702 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.702 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.702 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.702 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.702 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:09:36.702 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.702 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.702 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.702 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.702 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.702 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.702 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.703 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:09:36.703 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.703 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.703 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.703 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.703 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.703 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.703 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.703 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:09:36.703 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.703 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.703 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.703 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.703 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.704 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.704 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.704 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:09:36.704 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.704 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.704 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.704 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.704 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.704 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.704 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.704 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:09:36.704 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.704 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.705 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.705 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.705 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.705 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.705 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.705 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:09:36.705 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.705 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.705 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.705 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.705 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.705 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.705 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.705 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:09:36.706 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.706 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.706 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.706 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.706 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.706 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.706 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.706 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:09:36.706 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.706 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.706 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.706 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.706 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.706 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.707 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.707 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:09:36.707 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.707 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.707 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.707 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.707 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.707 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.707 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.707 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:09:36.707 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.707 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.707 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.707 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.708 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.708 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.708 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.708 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:09:36.708 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.708 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:09:36.708 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:09:36.708 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.708 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-22T03:09:36.708 INFO:teuthology.orchestra.run.smithi196.stdout: "type": "client_request", 2024-09-22T03:09:36.708 INFO:teuthology.orchestra.run.smithi196.stdout: "id": 19, 2024-09-22T03:09:36.708 INFO:teuthology.orchestra.run.smithi196.stdout: "detail": { 2024-09-22T03:09:36.708 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.709 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:09:36.709 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:09:35.894922+0000" 2024-09-22T03:09:36.709 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.709 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.709 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:09:36.709 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:09:35.894926+0000" 2024-09-22T03:09:36.709 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.709 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.709 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.709 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.709 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.709 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.709 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:09:36.709 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:09:36.003644+0000" 2024-09-22T03:09:36.710 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.710 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.710 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.710 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.710 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.710 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.710 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:09:36.710 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.710 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.710 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.710 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.710 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.710 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.711 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.711 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:09:36.711 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:09:36.004003+0000" 2024-09-22T03:09:36.711 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.711 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.711 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.711 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.711 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.711 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.711 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:09:36.711 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:09:36.004707+0000", 2024-09-22T03:09:36.711 INFO:teuthology.orchestra.run.smithi196.stdout: "blocker": { 2024-09-22T03:09:36.711 INFO:teuthology.orchestra.run.smithi196.stdout: "op_type": "PerShardPipeline::create_or_wait_pg", 2024-09-22T03:09:36.712 INFO:teuthology.orchestra.run.smithi196.stdout: "detail": { 2024-09-22T03:09:36.712 INFO:teuthology.orchestra.run.smithi196.stdout: "waiting": 5, 2024-09-22T03:09:36.712 INFO:teuthology.orchestra.run.smithi196.stdout: "held_by_operation_id": 17 2024-09-22T03:09:36.712 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:09:36.712 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:09:36.712 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.712 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.712 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.712 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.712 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.712 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.712 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:09:36.712 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.713 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.713 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.713 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.713 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.713 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.713 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.713 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:09:36.713 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.713 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.713 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.713 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:09:36.713 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.713 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.713 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.713 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.714 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.714 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.714 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.714 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:09:36.714 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.714 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.714 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.714 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.714 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.714 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.714 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.714 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:09:36.714 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.715 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.715 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.715 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.715 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.715 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.715 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.715 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:09:36.715 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.715 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.715 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.715 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.715 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.715 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.716 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.716 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:09:36.716 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.716 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.716 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.716 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.716 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.716 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.716 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.716 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:09:36.716 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.716 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.716 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.717 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.717 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.717 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.717 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.717 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:09:36.717 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.717 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.717 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.717 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.717 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.717 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.717 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.717 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:09:36.718 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.718 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.718 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.718 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.718 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.718 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.718 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.718 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:09:36.718 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.718 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.718 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.718 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.718 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.718 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.719 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.719 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:09:36.719 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.719 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.719 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.719 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.719 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.719 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.719 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.719 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:09:36.719 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.719 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.719 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.719 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.720 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.720 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.720 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.720 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:09:36.720 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.720 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.720 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.720 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.720 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.720 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.720 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.720 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:09:36.720 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.721 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.721 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.721 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.721 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.721 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.721 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.721 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:09:36.721 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.721 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:09:36.721 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:09:36.721 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.721 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-22T03:09:36.721 INFO:teuthology.orchestra.run.smithi196.stdout: "type": "client_request", 2024-09-22T03:09:36.721 INFO:teuthology.orchestra.run.smithi196.stdout: "id": 21, 2024-09-22T03:09:36.722 INFO:teuthology.orchestra.run.smithi196.stdout: "detail": { 2024-09-22T03:09:36.722 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.722 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:09:36.722 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:09:35.895436+0000" 2024-09-22T03:09:36.722 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.722 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.722 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:09:36.722 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:09:35.895443+0000" 2024-09-22T03:09:36.722 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.722 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.722 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.722 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.723 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.723 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.723 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:09:36.723 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:09:36.004368+0000" 2024-09-22T03:09:36.723 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.723 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.723 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.723 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.723 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.723 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.723 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:09:36.723 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.723 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.724 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.724 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.724 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.724 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.724 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.724 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:09:36.724 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:09:36.004470+0000" 2024-09-22T03:09:36.724 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.724 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.724 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.724 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.724 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.724 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.724 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:09:36.724 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:09:36.006763+0000", 2024-09-22T03:09:36.725 INFO:teuthology.orchestra.run.smithi196.stdout: "blocker": { 2024-09-22T03:09:36.725 INFO:teuthology.orchestra.run.smithi196.stdout: "op_type": "PerShardPipeline::create_or_wait_pg", 2024-09-22T03:09:36.725 INFO:teuthology.orchestra.run.smithi196.stdout: "detail": { 2024-09-22T03:09:36.725 INFO:teuthology.orchestra.run.smithi196.stdout: "waiting": 5, 2024-09-22T03:09:36.725 INFO:teuthology.orchestra.run.smithi196.stdout: "held_by_operation_id": 17 2024-09-22T03:09:36.725 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:09:36.725 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:09:36.725 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.725 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.725 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.725 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.725 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.725 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.725 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:09:36.726 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.726 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.726 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.726 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.726 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.726 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.726 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.726 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:09:36.726 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.726 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.726 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.726 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:09:36.726 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.726 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.727 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.727 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.727 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.727 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.727 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.727 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:09:36.727 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.727 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.727 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.727 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.727 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.727 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.727 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.728 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:09:36.728 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.728 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.728 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.728 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.728 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.728 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.728 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.728 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:09:36.728 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.728 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.728 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.728 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.728 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.729 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.729 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.729 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:09:36.729 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.729 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.729 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.729 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.729 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.729 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.729 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.729 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:09:36.729 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.729 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.729 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.730 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.730 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.730 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.730 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.730 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:09:36.730 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.730 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.730 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.730 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.730 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.730 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.730 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.730 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:09:36.730 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.731 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.731 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.731 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.731 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.731 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.731 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.731 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:09:36.731 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.731 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.731 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.731 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.731 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.731 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.732 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.732 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:09:36.732 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.732 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.732 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.732 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.732 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.732 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.732 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.732 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:09:36.732 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.732 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.732 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.732 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.733 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.733 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.733 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.733 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:09:36.733 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.733 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.733 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.733 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.733 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.733 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.733 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.733 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:09:36.733 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.733 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.733 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.734 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.734 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.734 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.734 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.734 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:09:36.734 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.734 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:09:36.734 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:09:36.734 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.734 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-22T03:09:36.734 INFO:teuthology.orchestra.run.smithi196.stdout: "type": "client_request", 2024-09-22T03:09:36.734 INFO:teuthology.orchestra.run.smithi196.stdout: "id": 20, 2024-09-22T03:09:36.734 INFO:teuthology.orchestra.run.smithi196.stdout: "detail": { 2024-09-22T03:09:36.734 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.735 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:09:36.735 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:09:35.895202+0000" 2024-09-22T03:09:36.735 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.735 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.735 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:09:36.735 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:09:35.895209+0000" 2024-09-22T03:09:36.735 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.735 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.735 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.735 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.735 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.735 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.735 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:09:36.735 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:09:36.004182+0000" 2024-09-22T03:09:36.736 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.736 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.736 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.736 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.736 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.736 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.736 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:09:36.736 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.736 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.736 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.736 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.736 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.736 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.736 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.737 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:09:36.737 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:09:36.004296+0000" 2024-09-22T03:09:36.737 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.737 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.737 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.737 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.737 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.737 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.737 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:09:36.737 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:09:36.006721+0000", 2024-09-22T03:09:36.737 INFO:teuthology.orchestra.run.smithi196.stdout: "blocker": { 2024-09-22T03:09:36.737 INFO:teuthology.orchestra.run.smithi196.stdout: "op_type": "PerShardPipeline::create_or_wait_pg", 2024-09-22T03:09:36.737 INFO:teuthology.orchestra.run.smithi196.stdout: "detail": { 2024-09-22T03:09:36.738 INFO:teuthology.orchestra.run.smithi196.stdout: "waiting": 5, 2024-09-22T03:09:36.738 INFO:teuthology.orchestra.run.smithi196.stdout: "held_by_operation_id": 17 2024-09-22T03:09:36.738 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:09:36.738 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:09:36.738 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.738 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.738 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.738 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.738 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.738 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.738 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:09:36.738 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.738 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.739 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.739 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.739 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.739 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.739 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.739 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:09:36.739 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.739 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.739 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.739 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:09:36.739 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.739 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.739 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.739 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.740 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.740 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.740 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.740 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:09:36.740 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.740 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.740 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.740 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.740 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.740 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.740 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.740 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:09:36.740 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.740 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.741 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.741 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.741 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.741 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.741 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.741 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:09:36.741 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.741 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.741 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.741 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.741 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.741 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.741 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.741 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:09:36.742 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.742 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.742 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.742 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.742 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.742 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.742 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.742 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:09:36.742 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.742 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.742 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.742 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.742 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.742 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.743 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.743 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:09:36.743 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.743 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.743 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.743 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.743 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.743 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.743 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.743 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:09:36.743 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.743 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.743 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.744 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.744 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.744 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.744 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.744 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:09:36.744 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.744 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.744 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.744 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.744 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.744 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.744 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.744 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:09:36.744 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.745 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.745 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.745 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.745 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.745 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.745 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.745 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:09:36.745 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.745 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.745 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.745 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.745 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.745 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.745 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.746 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:09:36.746 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.746 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.746 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.746 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.746 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.746 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.746 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.746 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:09:36.746 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.746 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.746 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.746 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.747 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.747 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.747 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.747 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:09:36.747 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.747 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:09:36.747 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:09:36.747 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.747 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-22T03:09:36.747 INFO:teuthology.orchestra.run.smithi196.stdout: "type": "client_request", 2024-09-22T03:09:36.747 INFO:teuthology.orchestra.run.smithi196.stdout: "id": 22, 2024-09-22T03:09:36.747 INFO:teuthology.orchestra.run.smithi196.stdout: "detail": { 2024-09-22T03:09:36.747 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.747 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:09:36.748 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:09:35.895839+0000" 2024-09-22T03:09:36.748 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.748 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.748 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:09:36.748 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:09:35.895846+0000" 2024-09-22T03:09:36.748 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.748 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.748 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.748 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.748 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.748 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.748 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:09:36.748 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:09:36.004801+0000" 2024-09-22T03:09:36.748 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.749 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.749 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.749 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.749 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.749 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.749 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:09:36.749 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.749 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.749 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.749 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.749 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.749 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.749 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.750 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:09:36.750 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:09:36.004899+0000" 2024-09-22T03:09:36.750 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.750 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.750 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.750 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.750 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.750 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.750 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:09:36.750 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:09:36.006800+0000", 2024-09-22T03:09:36.750 INFO:teuthology.orchestra.run.smithi196.stdout: "blocker": { 2024-09-22T03:09:36.750 INFO:teuthology.orchestra.run.smithi196.stdout: "op_type": "PerShardPipeline::create_or_wait_pg", 2024-09-22T03:09:36.750 INFO:teuthology.orchestra.run.smithi196.stdout: "detail": { 2024-09-22T03:09:36.750 INFO:teuthology.orchestra.run.smithi196.stdout: "waiting": 5, 2024-09-22T03:09:36.751 INFO:teuthology.orchestra.run.smithi196.stdout: "held_by_operation_id": 17 2024-09-22T03:09:36.751 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:09:36.751 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:09:36.751 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.751 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.751 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.751 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.751 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.751 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.751 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:09:36.751 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.751 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.751 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.751 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.752 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.752 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.752 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.752 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:09:36.752 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.752 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.752 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.752 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:09:36.752 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.752 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.752 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.752 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.752 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.753 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.753 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.753 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:09:36.753 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.753 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.753 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.753 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.753 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.753 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.753 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.753 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:09:36.753 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.753 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.753 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.754 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.754 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.754 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.754 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.754 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:09:36.754 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.754 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.754 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.754 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.754 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.754 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.754 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.754 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:09:36.754 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.755 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.755 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.755 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.755 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.755 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.755 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.755 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:09:36.755 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.755 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.755 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.755 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.755 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.755 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.755 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.756 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:09:36.756 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.756 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.756 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.756 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.756 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.756 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.756 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.756 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:09:36.756 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.756 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.756 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.756 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.756 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.756 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.757 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.757 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:09:36.757 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.757 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.757 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.757 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.757 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.757 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.757 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.757 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:09:36.757 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.757 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.757 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.757 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.758 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.758 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.758 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.758 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:09:36.758 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.758 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.758 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.758 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.758 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.758 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.758 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.758 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:09:36.758 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.759 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.759 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.759 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.759 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.759 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.759 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:09:36.759 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:09:36.759 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.759 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.759 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.759 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:36.759 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.759 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:09:36.760 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:09:36.760 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:09:36.760 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:36.760 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:09:36.760 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:09:36.760 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:09:36.760 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-22T03:09:36.760 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-09-22T03:09:37.011 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-22T03:09:37.140 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:09:37.137+0000 7f5e38397640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:09:37.143 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:09:37.142+0000 7f5e38397640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:09:37.148 INFO:teuthology.orchestra.run.smithi153.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-22T03:09:37.242 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-22T03:09:37.242 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-22T03:09:37.543 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 81 16 3471 3455 0.333193 0.0966797 0.051987 0.375015 2024-09-22T03:09:38.544 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 82 16 3846 3830 0.364853 2.92969 0.0123424 0.342554 2024-09-22T03:09:39.244 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-22T03:09:39.368 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:09:39.365+0000 7f5336f97640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:09:39.371 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:09:39.370+0000 7f5336f97640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:09:39.375 INFO:teuthology.orchestra.run.smithi153.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-22T03:09:39.544 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 83 16 4224 4208 0.396032 2.95312 0.0766021 0.315537 2024-09-22T03:09:39.559 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-22T03:09:39.559 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-22T03:09:40.544 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 84 16 4569 4553 0.423401 2.69531 0.0209438 0.295116 2024-09-22T03:09:41.544 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 85 16 4938 4922 0.452331 2.88281 0.0263719 0.276217 2024-09-22T03:09:41.560 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-22T03:09:41.676 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:09:41.672+0000 7f7dce997640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:09:41.679 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:09:41.677+0000 7f7dce997640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:09:41.683 INFO:teuthology.orchestra.run.smithi153.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-22T03:09:41.762 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [0] live_osds: [1, 2, 3] 2024-09-22T03:09:41.762 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-22T03:09:41.762 INFO:tasks.thrashosds.thrasher:Reviving osd 0 2024-09-22T03:09:41.762 INFO:tasks.ceph.osd.0:Restarting daemon 2024-09-22T03:09:41.762 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2>> /var/log/ceph/ceph-osd.0.log 2024-09-22T03:09:41.765 INFO:tasks.ceph.osd.0:Started 2024-09-22T03:09:41.765 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-22T03:09:41.797 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-22T03:09:41.798 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-22T03:09:41.879 INFO:teuthology.orchestra.run.smithi153.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-09-22T03:09:41.882 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-22T03:09:41.882 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-0, ['dump_ops_in_flight'] 2024-09-22T03:09:42.544 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 86 16 5273 5257 0.4775 2.61719 0.033311 0.261706 2024-09-22T03:09:43.544 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 87 16 5586 5570 0.500115 2.44531 0.0251701 0.24989 2024-09-22T03:09:43.799 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-22T03:09:43.918 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:09:43.913+0000 7f06fe0d7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:09:43.921 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:09:43.919+0000 7f06fe0d7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:09:43.926 INFO:teuthology.orchestra.run.smithi153.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-22T03:09:44.002 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-22T03:09:44.002 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-22T03:09:44.544 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 88 16 5944 5928 0.526211 2.79688 0.0633831 0.237452 2024-09-22T03:09:45.544 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 89 16 6268 6252 0.548736 2.53125 0.0496009 0.227705 2024-09-22T03:09:46.003 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-22T03:09:46.143 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:09:46.138+0000 7fd333f17640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:09:46.149 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:09:46.147+0000 7fd333f17640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:09:46.157 INFO:teuthology.orchestra.run.smithi153.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-22T03:09:46.333 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-22T03:09:46.333 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-22T03:09:46.544 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 90 13 6621 6608 0.573538 2.78125 0.0404446 0.217877 2024-09-22T03:09:46.883 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-22T03:09:46.982 INFO:teuthology.orchestra.run.smithi153.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-09-22T03:09:46.984 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-22T03:09:46.985 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-0, ['dump_ops_in_flight'] 2024-09-22T03:09:47.544 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:Total time run: 90.0299 2024-09-22T03:09:47.544 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:Total writes made: 6621 2024-09-22T03:09:47.545 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:Write size: 8192 2024-09-22T03:09:47.545 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:Object size: 65536 2024-09-22T03:09:47.545 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:Bandwidth (MB/sec): 0.574549 2024-09-22T03:09:47.545 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:Stddev Bandwidth: 1.12347 2024-09-22T03:09:47.545 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:Max bandwidth (MB/sec): 3.13281 2024-09-22T03:09:47.545 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:Min bandwidth (MB/sec): 0 2024-09-22T03:09:47.545 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:Average IOPS: 73 2024-09-22T03:09:47.545 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:Stddev IOPS: 143.809 2024-09-22T03:09:47.545 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:Max IOPS: 401 2024-09-22T03:09:47.545 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:Min IOPS: 0 2024-09-22T03:09:47.545 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:Average Latency(s): 0.217529 2024-09-22T03:09:47.545 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:Stddev Latency(s): 3.14197 2024-09-22T03:09:47.545 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:Max latency(s): 71.9939 2024-09-22T03:09:47.545 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:Min latency(s): 0.00729567 2024-09-22T03:09:47.558 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:Cleaning up (deleting benchmark objects) 2024-09-22T03:09:48.335 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-22T03:09:48.473 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:09:48.470+0000 7f1f84197640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:09:48.476 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:09:48.475+0000 7f1f84197640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:09:48.481 INFO:teuthology.orchestra.run.smithi153.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-22T03:09:48.676 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-22T03:09:48.677 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-22T03:09:50.678 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-22T03:09:50.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:09:50.777+0000 7f7c9e62c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:09:50.783 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:09:50.782+0000 7f7c9e62c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:09:51.986 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-22T03:09:52.142 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-22T03:09:52.143 INFO:teuthology.orchestra.run.smithi153.stdout: "ops_in_flight": [ 2024-09-22T03:09:52.143 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-22T03:09:52.143 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "background_recovery_sub", 2024-09-22T03:09:52.143 INFO:teuthology.orchestra.run.smithi153.stdout: "id": 16777396, 2024-09-22T03:09:52.143 INFO:teuthology.orchestra.run.smithi153.stdout: "detail": {} 2024-09-22T03:09:52.143 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:09:52.143 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-22T03:09:52.143 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "client_request", 2024-09-22T03:09:52.143 INFO:teuthology.orchestra.run.smithi153.stdout: "id": 129, 2024-09-22T03:09:52.143 INFO:teuthology.orchestra.run.smithi153.stdout: "detail": { 2024-09-22T03:09:52.143 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:09:52.144 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:09:52.144 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:09:51.482307+0000" 2024-09-22T03:09:52.144 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:09:52.144 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:09:52.144 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:09:52.144 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:09:51.482318+0000" 2024-09-22T03:09:52.144 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:09:52.144 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:09:52.144 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:52.144 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:52.144 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:09:52.144 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:09:52.144 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:09:52.145 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:09:51.482790+0000" 2024-09-22T03:09:52.145 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:09:52.145 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:09:52.145 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:52.145 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:52.145 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:09:52.145 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:09:52.145 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:09:52.145 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:52.145 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:09:52.145 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:09:52.145 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:52.145 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:52.145 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:09:52.146 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:09:52.146 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:09:52.146 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:09:51.482968+0000" 2024-09-22T03:09:52.146 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:09:52.146 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:09:52.146 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:52.146 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:52.146 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:09:52.146 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:09:52.146 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:09:52.146 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:09:51.483085+0000" 2024-09-22T03:09:52.146 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:09:52.146 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:09:52.147 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:52.147 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:52.147 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:09:52.147 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:09:52.147 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:09:52.147 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:52.147 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:09:52.147 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:09:52.147 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:52.147 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:52.147 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:09:52.147 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:09:52.147 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:09:52.148 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:09:51.483945+0000" 2024-09-22T03:09:52.148 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:09:52.148 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:09:52.148 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:09:52.148 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:52.148 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:09:52.148 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:09:52.148 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:52.148 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:52.148 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:09:52.148 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:09:52.148 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:09:52.148 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:52.148 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:09:52.148 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:09:52.149 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:52.149 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:52.149 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:09:52.149 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:09:52.149 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:09:52.149 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:52.149 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:09:52.149 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:09:52.149 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:52.149 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:52.149 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:09:52.149 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:09:52.149 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:09:52.149 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:52.150 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:09:52.150 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:09:52.150 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:52.150 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:52.150 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:09:52.150 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:09:52.150 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:09:52.150 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:52.150 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:09:52.150 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:09:52.150 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:52.150 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:52.150 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:09:52.151 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:09:52.151 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:09:52.151 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:52.151 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:09:52.151 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:09:52.151 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:52.151 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:52.151 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:09:52.151 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:09:52.151 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:09:52.151 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:52.151 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:09:52.151 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:09:52.151 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:52.152 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:52.152 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:09:52.152 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:09:52.152 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:09:52.152 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:52.152 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:09:52.152 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:09:52.152 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:52.152 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:52.152 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:09:52.152 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:09:52.152 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:09:52.152 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:52.152 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:09:52.152 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:09:52.153 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:52.153 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:52.153 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:09:52.153 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:09:52.153 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:09:52.153 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:52.153 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:09:52.153 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:09:52.153 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:52.153 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:52.153 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:09:52.153 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:09:52.153 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:09:52.153 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:52.153 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:09:52.154 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:09:52.154 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:52.154 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:52.154 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:09:52.154 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:09:52.154 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:09:52.154 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:52.154 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:09:52.154 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:09:52.154 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:52.154 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:52.154 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:09:52.154 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:09:52.154 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:09:52.155 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:52.155 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:09:52.155 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:09:52.155 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:52.155 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:52.155 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:09:52.155 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:09:52.155 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:09:52.155 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:52.155 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:09:52.155 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:09:52.155 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:09:52.155 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-22T03:09:52.155 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "client_request", 2024-09-22T03:09:52.155 INFO:teuthology.orchestra.run.smithi153.stdout: "id": 131, 2024-09-22T03:09:52.155 INFO:teuthology.orchestra.run.smithi153.stdout: "detail": { 2024-09-22T03:09:52.156 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:09:52.156 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:09:52.156 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:09:51.486476+0000" 2024-09-22T03:09:52.156 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:09:52.156 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:09:52.156 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:09:52.156 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:09:51.486480+0000" 2024-09-22T03:09:52.156 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:09:52.156 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:09:52.156 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:52.156 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:52.156 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:09:52.156 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:09:52.156 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:09:52.156 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:09:51.486613+0000" 2024-09-22T03:09:52.156 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:09:52.157 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:09:52.157 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:52.157 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:52.157 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:09:52.157 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:09:52.157 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:09:52.157 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:52.157 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:09:52.157 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:09:52.157 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:52.157 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:52.157 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:09:52.157 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:09:52.157 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:09:52.158 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:09:51.486691+0000" 2024-09-22T03:09:52.158 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:09:52.158 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:09:52.158 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:52.158 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:52.158 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:09:52.158 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:09:52.158 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:09:52.158 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:09:51.486798+0000" 2024-09-22T03:09:52.158 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:09:52.158 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:09:52.158 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:52.158 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:52.158 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:09:52.158 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:09:52.159 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:09:52.159 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:52.159 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:09:52.159 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:09:52.159 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:52.159 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:52.159 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:09:52.159 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:09:52.159 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:09:52.159 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:09:51.487557+0000" 2024-09-22T03:09:52.159 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:09:52.159 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:09:52.159 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:09:52.160 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:52.160 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:09:52.160 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:09:52.160 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:52.160 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:52.160 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:09:52.160 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:09:52.160 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:09:52.160 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:52.160 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:09:52.160 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:09:52.160 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:52.160 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:52.161 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:09:52.161 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:09:52.161 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:09:52.161 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:52.161 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:09:52.161 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:09:52.161 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:52.161 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:52.161 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:09:52.161 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:09:52.161 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:09:52.161 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:52.161 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:09:52.161 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:09:52.162 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:52.162 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:52.162 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:09:52.162 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:09:52.162 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:09:52.162 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:52.162 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:09:52.162 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:09:52.162 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:52.162 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:52.162 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:09:52.162 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:09:52.162 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:09:52.163 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:52.163 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:09:52.163 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:09:52.163 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:52.163 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:52.163 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:09:52.163 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:09:52.163 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:09:52.163 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:52.163 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:09:52.163 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:09:52.163 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:52.163 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:52.163 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:09:52.164 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:09:52.164 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:09:52.164 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:52.164 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:09:52.164 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:09:52.164 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:52.164 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:52.164 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:09:52.164 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:09:52.164 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:09:52.164 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:52.164 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:09:52.164 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:09:52.165 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:52.165 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:52.165 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:09:52.165 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:09:52.165 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:09:52.165 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:52.165 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:09:52.165 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:09:52.165 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:52.165 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:52.165 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:09:52.165 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:09:52.165 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:09:52.166 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:52.166 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:09:52.166 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:09:52.166 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:52.166 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:52.166 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:09:52.166 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:09:52.166 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:09:52.166 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:52.166 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:09:52.166 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:09:52.166 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:52.166 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:52.166 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:09:52.167 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:09:52.167 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:09:52.167 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:52.167 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:09:52.167 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:09:52.167 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:52.167 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:52.167 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:09:52.167 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:09:52.167 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:09:52.167 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:52.167 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:09:52.167 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:09:52.168 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:09:52.168 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-22T03:09:52.168 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "replicated_request", 2024-09-22T03:09:52.168 INFO:teuthology.orchestra.run.smithi153.stdout: "id": 33554628, 2024-09-22T03:09:52.168 INFO:teuthology.orchestra.run.smithi153.stdout: "detail": { 2024-09-22T03:09:52.168 INFO:teuthology.orchestra.run.smithi153.stdout: "RepRequest": { 2024-09-22T03:09:52.168 INFO:teuthology.orchestra.run.smithi153.stdout: "reqid": "client.4243.0:7446", 2024-09-22T03:09:52.168 INFO:teuthology.orchestra.run.smithi153.stdout: "pgid": "3.1", 2024-09-22T03:09:52.168 INFO:teuthology.orchestra.run.smithi153.stdout: "map_epoch": 29, 2024-09-22T03:09:52.168 INFO:teuthology.orchestra.run.smithi153.stdout: "min_epoch": 28, 2024-09-22T03:09:52.168 INFO:teuthology.orchestra.run.smithi153.stdout: "oid": "3:89bdad79:::benchmark_data_smithi153_30331_object822:head", 2024-09-22T03:09:52.168 INFO:teuthology.orchestra.run.smithi153.stdout: "from": "3" 2024-09-22T03:09:52.168 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:09:52.168 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:09:52.169 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:09:52.169 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-22T03:09:52.169 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "background_recovery_sub", 2024-09-22T03:09:52.169 INFO:teuthology.orchestra.run.smithi153.stdout: "id": 33554624, 2024-09-22T03:09:52.169 INFO:teuthology.orchestra.run.smithi153.stdout: "detail": {} 2024-09-22T03:09:52.169 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:09:52.169 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-22T03:09:52.169 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "background_recovery_sub", 2024-09-22T03:09:52.169 INFO:teuthology.orchestra.run.smithi153.stdout: "id": 33554627, 2024-09-22T03:09:52.169 INFO:teuthology.orchestra.run.smithi153.stdout: "detail": {} 2024-09-22T03:09:52.169 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:09:52.169 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-22T03:09:52.169 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "background_recovery_sub", 2024-09-22T03:09:52.170 INFO:teuthology.orchestra.run.smithi153.stdout: "id": 16777397, 2024-09-22T03:09:52.170 INFO:teuthology.orchestra.run.smithi153.stdout: "detail": {} 2024-09-22T03:09:52.170 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:09:52.170 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-22T03:09:52.170 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "client_request", 2024-09-22T03:09:52.170 INFO:teuthology.orchestra.run.smithi153.stdout: "id": 39, 2024-09-22T03:09:52.170 INFO:teuthology.orchestra.run.smithi153.stdout: "detail": { 2024-09-22T03:09:52.170 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:09:52.170 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:09:52.170 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:09:49.291039+0000" 2024-09-22T03:09:52.170 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:09:52.170 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:09:52.170 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:09:52.171 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:09:49.291062+0000" 2024-09-22T03:09:52.171 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:09:52.171 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:09:52.171 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:52.171 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:52.171 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:09:52.171 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:09:52.171 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:09:52.171 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:09:49.291166+0000" 2024-09-22T03:09:52.171 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:09:52.171 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:09:52.171 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:52.171 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:52.171 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:09:52.172 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:09:52.172 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:09:52.172 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:52.172 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:09:52.172 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:09:52.172 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:52.172 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:52.172 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:09:52.172 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:09:52.172 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:09:52.172 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:09:49.291235+0000" 2024-09-22T03:09:52.172 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:09:52.172 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:09:52.172 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:52.173 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:52.173 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:09:52.173 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:09:52.173 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:09:52.173 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:09:49.291799+0000" 2024-09-22T03:09:52.173 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:09:52.173 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:09:52.173 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:52.173 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:52.173 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:09:52.173 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:09:52.173 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:09:52.173 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:52.174 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:09:52.174 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:09:52.174 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:52.174 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:52.174 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:09:52.174 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:09:52.174 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:09:52.174 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:09:50.215287+0000" 2024-09-22T03:09:52.174 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:09:52.174 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:09:52.174 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:09:52.174 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:09:49.292055+0000" 2024-09-22T03:09:52.174 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:09:52.175 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:09:52.175 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:52.175 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:52.175 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:09:52.175 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:09:52.175 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:09:52.175 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:52.175 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:09:52.175 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:09:52.175 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:52.175 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:52.175 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:09:52.175 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:09:52.175 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:09:52.176 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:09:49.292206+0000" 2024-09-22T03:09:52.176 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:09:52.176 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:09:52.176 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:52.176 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:52.176 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:09:52.176 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:09:52.176 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:09:52.176 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:09:49.292305+0000" 2024-09-22T03:09:52.176 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:09:52.176 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:09:52.176 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:52.176 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:52.177 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:09:52.177 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:09:52.177 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:09:52.177 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:52.177 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:09:52.177 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:09:52.177 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:52.177 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:52.177 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:09:52.177 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:09:52.177 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:09:52.177 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:52.177 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:09:52.177 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:09:52.178 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:52.178 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:52.178 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:09:52.178 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:09:52.178 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:09:52.178 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:52.178 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:09:52.178 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:09:52.178 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:52.178 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:52.178 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:09:52.178 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:09:52.178 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:09:52.179 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:52.179 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:09:52.179 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:09:52.179 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:52.179 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:52.179 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:09:52.179 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:09:52.179 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:09:52.179 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:52.179 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:09:52.179 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:09:52.179 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:52.179 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:52.179 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:09:52.180 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:09:52.180 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:09:52.180 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:52.180 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:09:52.180 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:09:52.180 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:52.180 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:52.180 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:09:52.180 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:09:52.180 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:09:52.180 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:52.180 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:09:52.180 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:09:52.181 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:52.181 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:52.181 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:09:52.181 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:09:52.181 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:09:52.181 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:52.181 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:09:52.181 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:09:52.181 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:52.181 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:52.181 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:09:52.181 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:09:52.181 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:09:52.182 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:52.182 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:09:52.182 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:09:52.182 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:09:52.182 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:52.182 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:09:52.182 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:09:52.182 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:09:52.182 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:09:52.182 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:09:52.182 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:09:52.182 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:09:52.182 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-22T03:09:52.182 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "background_recovery_sub", 2024-09-22T03:09:52.183 INFO:teuthology.orchestra.run.smithi153.stdout: "id": 245, 2024-09-22T03:09:52.183 INFO:teuthology.orchestra.run.smithi153.stdout: "detail": {} 2024-09-22T03:09:52.183 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:09:52.183 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-22T03:09:52.183 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "background_recovery_sub", 2024-09-22T03:09:52.183 INFO:teuthology.orchestra.run.smithi153.stdout: "id": 16777394, 2024-09-22T03:09:52.183 INFO:teuthology.orchestra.run.smithi153.stdout: "detail": {} 2024-09-22T03:09:52.183 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:09:52.183 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-22T03:09:52.183 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "background_recovery_sub", 2024-09-22T03:09:52.183 INFO:teuthology.orchestra.run.smithi153.stdout: "id": 33554629, 2024-09-22T03:09:52.183 INFO:teuthology.orchestra.run.smithi153.stdout: "detail": {} 2024-09-22T03:09:52.183 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:09:52.184 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-22T03:09:52.184 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-22T03:09:52.284 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:Removed 828 objects 2024-09-22T03:09:52.284 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:Clean up completed and total clean up time :4.72602 2024-09-22T03:09:52.853 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-22T03:09:52.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:09:52.976+0000 7f4f2f8a9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:09:52.984 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:09:52.983+0000 7f4f2f8a9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:09:54.100 INFO:tasks.ceph.ceph_manager.ceph:removing pool_name unique_pool_0 2024-09-22T03:09:54.100 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm unique_pool_0 unique_pool_0 --yes-i-really-really-mean-it 2024-09-22T03:09:54.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:09:54.200+0000 7f5cf842c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:09:54.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:09:54.211+0000 7f5cf842c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:09:54.860 INFO:teuthology.orchestra.run.smithi153.stderr:pool 'unique_pool_0' removed 2024-09-22T03:09:54.880 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-09-22T03:09:54.881 INFO:tasks.radosbench:Beginning radosbench... 2024-09-22T03:09:54.881 INFO:tasks.ceph.ceph_manager.ceph:creating pool_name unique_pool_1 2024-09-22T03:09:54.881 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create unique_pool_1 16 2024-09-22T03:09:55.020 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:09:55.012+0000 7fb1b9997640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:09:55.024 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:09:55.022+0000 7fb1b9997640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:09:55.077 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-22T03:09:55.181 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:09:55.178+0000 7efcb3f97640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:09:55.184 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:09:55.182+0000 7efcb3f97640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:09:56.201 INFO:teuthology.orchestra.run.smithi153.stderr:pool 'unique_pool_1' created 2024-09-22T03:09:56.220 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool application enable unique_pool_1 rados --yes-i-really-mean-it || true 2024-09-22T03:09:56.344 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:09:56.333+0000 7ff276aa9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:09:56.347 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:09:56.345+0000 7ff276aa9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:09:57.186 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [1, 2, 3, 0] 2024-09-22T03:09:57.186 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-22T03:09:57.186 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-22T03:09:57.206 INFO:teuthology.orchestra.run.smithi153.stderr:enabled application 'rados' on pool 'unique_pool_1' 2024-09-22T03:09:57.259 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-22T03:09:57.340 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:09:57.332+0000 7fed9a458640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:09:57.345 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:09:57.343+0000 7fed9a458640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:09:57.404 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:09:57.401+0000 7f37316b2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:09:57.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:09:57.405+0000 7f37316b2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:09:57.529 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-22T03:09:57.530 INFO:teuthology.orchestra.run.smithi153.stderr:dumped all 2024-09-22T03:09:57.542 INFO:teuthology.orchestra.run.smithi153.stdout:{"pg_ready":true,"pg_map":{"version":84,"stamp":"2024-09-22T03:09:57.209207+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":139,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":60,"acting":60,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":142684,"kb_used_data":1244,"kb_used_omap":0,"kb_used_meta":141376,"kb_avail":377344676,"statfs":{"total":386547056640,"available":386400948224,"internally_reserved":0,"allocated":1273856,"data_stored":2150716,"data_compressed":89886,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":144769024},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-3186712,"num_objects":-50,"num_object_clones":0,"num_object_copies":-150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-229,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":-1,"num_read_kb":-1,"num_write":-7401,"num_write_kb":-52969,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-123,"num_bytes_recovered":-5431296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.012688"},"pg_stats":[{"pgid":"4.f","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-09-22T03:09:56.237476+0000","last_change":"2024-09-22T03:09:56.237476+0000","last_active":"2024-09-22T03:09:56.197496+0000","last_peered":"2024-09-22T03:09:56.197496+0000","last_clean":"2024-09-22T03:09:56.197496+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-22T03:09:56.237476+0000","last_undegraded":"2024-09-22T03:09:56.237476+0000","last_fullsized":"2024-09-22T03:09:56.237476+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:09:56.197496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:09:56.197496+0000","last_clean_scrub_stamp":"2024-09-22T03:09:56.197496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,3],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-09-22T03:09:56.229593+0000","last_change":"2024-09-22T03:09:56.229593+0000","last_active":"2024-09-22T03:09:56.197496+0000","last_peered":"2024-09-22T03:09:56.197496+0000","last_clean":"2024-09-22T03:09:56.197496+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-22T03:09:56.229593+0000","last_undegraded":"2024-09-22T03:09:56.229593+0000","last_fullsized":"2024-09-22T03:09:56.229593+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:09:56.197496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:09:56.197496+0000","last_clean_scrub_stamp":"2024-09-22T03:09:56.197496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,3],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-09-22T03:09:56.265241+0000","last_change":"2024-09-22T03:09:56.265241+0000","last_active":"2024-09-22T03:09:56.197496+0000","last_peered":"2024-09-22T03:09:56.197496+0000","last_clean":"2024-09-22T03:09:56.197496+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-22T03:09:56.265241+0000","last_undegraded":"2024-09-22T03:09:56.265241+0000","last_fullsized":"2024-09-22T03:09:56.265241+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:09:56.197496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:09:56.197496+0000","last_clean_scrub_stamp":"2024-09-22T03:09:56.197496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,2],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-22T03:09:56.197496+0000","last_change":"2024-09-22T03:09:56.197496+0000","last_active":"2024-09-22T03:09:56.197496+0000","last_peered":"2024-09-22T03:09:56.197496+0000","last_clean":"2024-09-22T03:09:56.197496+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-22T03:09:56.197496+0000","last_undegraded":"2024-09-22T03:09:56.197496+0000","last_fullsized":"2024-09-22T03:09:56.197496+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:09:56.197496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:09:56.197496+0000","last_clean_scrub_stamp":"2024-09-22T03:09:56.197496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-22T03:09:56.197496+0000","last_change":"2024-09-22T03:09:56.197496+0000","last_active":"2024-09-22T03:09:56.197496+0000","last_peered":"2024-09-22T03:09:56.197496+0000","last_clean":"2024-09-22T03:09:56.197496+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-22T03:09:56.197496+0000","last_undegraded":"2024-09-22T03:09:56.197496+0000","last_fullsized":"2024-09-22T03:09:56.197496+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:09:56.197496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:09:56.197496+0000","last_clean_scrub_stamp":"2024-09-22T03:09:56.197496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-09-22T03:09:56.253587+0000","last_change":"2024-09-22T03:09:56.253587+0000","last_active":"2024-09-22T03:09:56.197496+0000","last_peered":"2024-09-22T03:09:56.197496+0000","last_clean":"2024-09-22T03:09:56.197496+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-22T03:09:56.253587+0000","last_undegraded":"2024-09-22T03:09:56.253587+0000","last_fullsized":"2024-09-22T03:09:56.253587+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:09:56.197496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:09:56.197496+0000","last_clean_scrub_stamp":"2024-09-22T03:09:56.197496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,3],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-09-22T03:09:56.244046+0000","last_change":"2024-09-22T03:09:56.244046+0000","last_active":"2024-09-22T03:09:56.197496+0000","last_peered":"2024-09-22T03:09:56.197496+0000","last_clean":"2024-09-22T03:09:56.197496+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-22T03:09:56.244046+0000","last_undegraded":"2024-09-22T03:09:56.244046+0000","last_fullsized":"2024-09-22T03:09:56.244046+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:09:56.197496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:09:56.197496+0000","last_clean_scrub_stamp":"2024-09-22T03:09:56.197496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,3],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-09-22T03:09:56.237114+0000","last_change":"2024-09-22T03:09:56.237114+0000","last_active":"2024-09-22T03:09:56.197496+0000","last_peered":"2024-09-22T03:09:56.197496+0000","last_clean":"2024-09-22T03:09:56.197496+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-22T03:09:56.237114+0000","last_undegraded":"2024-09-22T03:09:56.237114+0000","last_fullsized":"2024-09-22T03:09:56.237114+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:09:56.197496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:09:56.197496+0000","last_clean_scrub_stamp":"2024-09-22T03:09:56.197496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,3],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-09-22T03:09:56.227707+0000","last_change":"2024-09-22T03:09:56.227707+0000","last_active":"2024-09-22T03:09:56.197496+0000","last_peered":"2024-09-22T03:09:56.197496+0000","last_clean":"2024-09-22T03:09:56.197496+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-22T03:09:56.227707+0000","last_undegraded":"2024-09-22T03:09:56.227707+0000","last_fullsized":"2024-09-22T03:09:56.227707+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:09:56.197496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:09:56.197496+0000","last_clean_scrub_stamp":"2024-09-22T03:09:56.197496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,3],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":38,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-22T03:09:54.892978+0000","last_change":"2024-09-22T03:09:51.214531+0000","last_active":"2024-09-22T03:09:54.892978+0000","last_peered":"2024-09-22T03:09:54.892978+0000","last_clean":"2024-09-22T03:09:54.892978+0000","last_became_active":"2024-09-22T03:09:50.269366+0000","last_became_peered":"2024-09-22T03:09:50.269366+0000","last_unstale":"2024-09-22T03:09:54.892978+0000","last_undegraded":"2024-09-22T03:09:54.892978+0000","last_fullsized":"2024-09-22T03:09:54.892978+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-09-22T03:09:56.212847+0000","last_change":"2024-09-22T03:09:56.212847+0000","last_active":"2024-09-22T03:09:56.197496+0000","last_peered":"2024-09-22T03:09:56.197496+0000","last_clean":"2024-09-22T03:09:56.197496+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-22T03:09:56.212847+0000","last_undegraded":"2024-09-22T03:09:56.212847+0000","last_fullsized":"2024-09-22T03:09:56.212847+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:09:56.197496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:09:56.197496+0000","last_clean_scrub_stamp":"2024-09-22T03:09:56.197496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,3],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":56,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-22T03:09:53.872006+0000","last_change":"2024-09-22T03:09:51.203613+0000","last_active":"2024-09-22T03:09:53.872006+0000","last_peered":"2024-09-22T03:09:53.872006+0000","last_clean":"2024-09-22T03:09:53.872006+0000","last_became_active":"2024-09-22T03:09:50.271057+0000","last_became_peered":"2024-09-22T03:09:50.271057+0000","last_unstale":"2024-09-22T03:09:53.872006+0000","last_undegraded":"2024-09-22T03:09:53.872006+0000","last_fullsized":"2024-09-22T03:09:53.872006+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-09-22T03:09:56.220586+0000","last_change":"2024-09-22T03:09:56.220586+0000","last_active":"2024-09-22T03:09:56.197496+0000","last_peered":"2024-09-22T03:09:56.197496+0000","last_clean":"2024-09-22T03:09:56.197496+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-22T03:09:56.220586+0000","last_undegraded":"2024-09-22T03:09:56.220586+0000","last_fullsized":"2024-09-22T03:09:56.220586+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:09:56.197496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:09:56.197496+0000","last_clean_scrub_stamp":"2024-09-22T03:09:56.197496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,3],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":62,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-22T03:09:56.208995+0000","last_change":"2024-09-22T03:09:49.221117+0000","last_active":"2024-09-22T03:09:56.208995+0000","last_peered":"2024-09-22T03:09:56.208995+0000","last_clean":"2024-09-22T03:09:56.208995+0000","last_became_active":"2024-09-22T03:09:36.978236+0000","last_became_peered":"2024-09-22T03:09:36.978236+0000","last_unstale":"2024-09-22T03:09:56.208995+0000","last_undegraded":"2024-09-22T03:09:56.208995+0000","last_fullsized":"2024-09-22T03:09:56.208995+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-22T03:09:56.197496+0000","last_change":"2024-09-22T03:09:56.197496+0000","last_active":"2024-09-22T03:09:56.197496+0000","last_peered":"2024-09-22T03:09:56.197496+0000","last_clean":"2024-09-22T03:09:56.197496+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-22T03:09:56.197496+0000","last_undegraded":"2024-09-22T03:09:56.197496+0000","last_fullsized":"2024-09-22T03:09:56.197496+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:09:56.197496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:09:56.197496+0000","last_clean_scrub_stamp":"2024-09-22T03:09:56.197496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":39,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-22T03:09:54.883110+0000","last_change":"2024-09-22T03:09:51.229549+0000","last_active":"2024-09-22T03:09:54.883110+0000","last_peered":"2024-09-22T03:09:54.883110+0000","last_clean":"2024-09-22T03:09:54.883110+0000","last_became_active":"2024-09-22T03:09:50.264602+0000","last_became_peered":"2024-09-22T03:09:50.264602+0000","last_unstale":"2024-09-22T03:09:54.883110+0000","last_undegraded":"2024-09-22T03:09:54.883110+0000","last_fullsized":"2024-09-22T03:09:54.883110+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"12'97","reported_seq":181,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-22T03:09:53.869969+0000","last_change":"2024-09-22T03:09:50.267908+0000","last_active":"2024-09-22T03:09:53.869969+0000","last_peered":"2024-09-22T03:09:53.869969+0000","last_clean":"2024-09-22T03:09:53.869969+0000","last_became_active":"2024-09-22T03:09:50.262052+0000","last_became_peered":"2024-09-22T03:09:50.262052+0000","last_unstale":"2024-09-22T03:09:53.869969+0000","last_undegraded":"2024-09-22T03:09:53.869969+0000","last_fullsized":"2024-09-22T03:09:53.869969+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-09-22T03:09:56.212787+0000","last_change":"2024-09-22T03:09:56.212787+0000","last_active":"2024-09-22T03:09:56.197496+0000","last_peered":"2024-09-22T03:09:56.197496+0000","last_clean":"2024-09-22T03:09:56.197496+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-22T03:09:56.212787+0000","last_undegraded":"2024-09-22T03:09:56.212787+0000","last_fullsized":"2024-09-22T03:09:56.212787+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:09:56.197496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:09:56.197496+0000","last_clean_scrub_stamp":"2024-09-22T03:09:56.197496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,3],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":78,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-22T03:09:56.208086+0000","last_change":"2024-09-22T03:09:51.197934+0000","last_active":"2024-09-22T03:09:56.208086+0000","last_peered":"2024-09-22T03:09:56.208086+0000","last_clean":"2024-09-22T03:09:56.208086+0000","last_became_active":"2024-09-22T03:09:50.257149+0000","last_became_peered":"2024-09-22T03:09:50.257149+0000","last_unstale":"2024-09-22T03:09:56.208086+0000","last_undegraded":"2024-09-22T03:09:56.208086+0000","last_fullsized":"2024-09-22T03:09:56.208086+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-22T03:09:56.197496+0000","last_change":"2024-09-22T03:09:56.197496+0000","last_active":"2024-09-22T03:09:56.197496+0000","last_peered":"2024-09-22T03:09:56.197496+0000","last_clean":"2024-09-22T03:09:56.197496+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-22T03:09:56.197496+0000","last_undegraded":"2024-09-22T03:09:56.197496+0000","last_fullsized":"2024-09-22T03:09:56.197496+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:09:56.197496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:09:56.197496+0000","last_clean_scrub_stamp":"2024-09-22T03:09:56.197496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":51,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-22T03:09:56.207371+0000","last_change":"2024-09-22T03:09:51.235909+0000","last_active":"2024-09-22T03:09:56.207371+0000","last_peered":"2024-09-22T03:09:56.207371+0000","last_clean":"2024-09-22T03:09:56.207371+0000","last_became_active":"2024-09-22T03:09:50.257567+0000","last_became_peered":"2024-09-22T03:09:50.257567+0000","last_unstale":"2024-09-22T03:09:56.207371+0000","last_undegraded":"2024-09-22T03:09:56.207371+0000","last_fullsized":"2024-09-22T03:09:56.207371+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-22T03:09:56.197496+0000","last_change":"2024-09-22T03:09:56.197496+0000","last_active":"2024-09-22T03:09:56.197496+0000","last_peered":"2024-09-22T03:09:56.197496+0000","last_clean":"2024-09-22T03:09:56.197496+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-22T03:09:56.197496+0000","last_undegraded":"2024-09-22T03:09:56.197496+0000","last_fullsized":"2024-09-22T03:09:56.197496+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:09:56.197496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:09:56.197496+0000","last_clean_scrub_stamp":"2024-09-22T03:09:56.197496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":70,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-22T03:09:56.209072+0000","last_change":"2024-09-22T03:09:49.206717+0000","last_active":"2024-09-22T03:09:56.209072+0000","last_peered":"2024-09-22T03:09:56.209072+0000","last_clean":"2024-09-22T03:09:56.209072+0000","last_became_active":"2024-09-22T03:09:36.954039+0000","last_became_peered":"2024-09-22T03:09:36.954039+0000","last_unstale":"2024-09-22T03:09:56.209072+0000","last_undegraded":"2024-09-22T03:09:56.209072+0000","last_fullsized":"2024-09-22T03:09:56.209072+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-09-22T03:09:56.219054+0000","last_change":"2024-09-22T03:09:56.219054+0000","last_active":"2024-09-22T03:09:56.197496+0000","last_peered":"2024-09-22T03:09:56.197496+0000","last_clean":"2024-09-22T03:09:56.197496+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-22T03:09:56.219054+0000","last_undegraded":"2024-09-22T03:09:56.219054+0000","last_fullsized":"2024-09-22T03:09:56.219054+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:09:56.197496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:09:56.197496+0000","last_clean_scrub_stamp":"2024-09-22T03:09:56.197496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,3],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":74,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-22T03:09:56.209606+0000","last_change":"2024-09-22T03:09:51.242005+0000","last_active":"2024-09-22T03:09:56.209606+0000","last_peered":"2024-09-22T03:09:56.209606+0000","last_clean":"2024-09-22T03:09:56.209606+0000","last_became_active":"2024-09-22T03:09:50.266815+0000","last_became_peered":"2024-09-22T03:09:50.266815+0000","last_unstale":"2024-09-22T03:09:56.209606+0000","last_undegraded":"2024-09-22T03:09:56.209606+0000","last_fullsized":"2024-09-22T03:09:56.209606+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":33,"acting":33,"num_store_stats":2},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1180736,"data_compressed":59924,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":25,"seq":107374182420,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":35740,"kb_used_data":268,"kb_used_omap":0,"kb_used_meta":35456,"kb_avail":94336100,"statfs":{"total":96636764160,"available":96600166400,"internally_reserved":0,"allocated":274432,"data_stored":94824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":36306944},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":19,"seq":81604378698,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31584,"kb_used_data":336,"kb_used_omap":0,"kb_used_meta":31232,"kb_avail":94340256,"statfs":{"total":96636764160,"available":96604422144,"internally_reserved":0,"allocated":344064,"data_stored":688796,"data_compressed":29962,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":31981568},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":10,"seq":42949673074,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":35480,"kb_used_data":328,"kb_used_omap":0,"kb_used_meta":35136,"kb_avail":94336360,"statfs":{"total":96636764160,"available":96600432640,"internally_reserved":0,"allocated":335872,"data_stored":685192,"data_compressed":29962,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":35979264},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":27,"seq":115964116997,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":39880,"kb_used_data":312,"kb_used_omap":0,"kb_used_meta":39552,"kb_avail":94331960,"statfs":{"total":96636764160,"available":96595927040,"internally_reserved":0,"allocated":319488,"data_stored":681904,"data_compressed":29962,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":40501248},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29962,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29962,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-22T03:09:58.229 DEBUG:teuthology.orchestra.run.smithi153:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 8192 --object-size 65536 -p unique_pool_1 bench 90 write' 2024-09-22T03:09:58.232 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-09-22T03:09:58.232 INFO:teuthology.orchestra.run:waiting for 3000 2024-09-22T03:09:58.295 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:hints = 1 2024-09-22T03:09:58.295 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:Maintaining 16 concurrent writes of 8192 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-09-22T03:09:58.295 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:Object prefix: benchmark_data_smithi153_32078 2024-09-22T03:09:58.295 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-22T03:09:58.295 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 0 0 0 0 0 0 - 0 2024-09-22T03:09:59.295 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 1 16 411 395 3.08576 3.08594 0.0583734 0.0394127 2024-09-22T03:09:59.474 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-22T03:09:59.642 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:09:59.638+0000 7f48f8557640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:09:59.647 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:09:59.645+0000 7f48f8557640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:10:00.295 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 2 16 777 761 2.97246 2.85938 0.0478905 0.041586 2024-09-22T03:10:01.295 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 3 16 1162 1146 2.98416 3.00781 0.0460777 0.0415622 2024-09-22T03:10:01.764 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-22T03:10:01.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:10:01.911+0000 7f8fe894e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:10:01.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:10:01.918+0000 7f8fe894e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:10:02.296 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 4 16 1517 1501 2.93143 2.77344 0.0780487 0.0423203 2024-09-22T03:10:02.544 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [1, 2, 3, 0] 2024-09-22T03:10:02.544 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-22T03:10:02.545 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-09-22T03:10:02.545 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-09-22T03:10:02.687 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:10:02.679+0000 7f33dcf57640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:10:02.691 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:10:02.689+0000 7f33dcf57640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:10:02.715 INFO:teuthology.orchestra.run.smithi153.stderr:no valid command found; 10 closest matches: 2024-09-22T03:10:02.715 INFO:teuthology.orchestra.run.smithi153.stderr:config get 2024-09-22T03:10:02.715 INFO:teuthology.orchestra.run.smithi153.stderr:config help 2024-09-22T03:10:02.715 INFO:teuthology.orchestra.run.smithi153.stderr:config set ... 2024-09-22T03:10:02.715 INFO:teuthology.orchestra.run.smithi153.stderr:config show 2024-09-22T03:10:02.715 INFO:teuthology.orchestra.run.smithi153.stderr:deep_scrub 2024-09-22T03:10:02.715 INFO:teuthology.orchestra.run.smithi153.stderr:dump_historic_ops 2024-09-22T03:10:02.715 INFO:teuthology.orchestra.run.smithi153.stderr:dump_historic_slow_ops 2024-09-22T03:10:02.715 INFO:teuthology.orchestra.run.smithi153.stderr:dump_metrics [] 2024-09-22T03:10:02.715 INFO:teuthology.orchestra.run.smithi153.stderr:dump_ops_in_flight 2024-09-22T03:10:02.715 INFO:teuthology.orchestra.run.smithi153.stderr:dump_pgstate_history 2024-09-22T03:10:02.716 INFO:teuthology.orchestra.run.smithi153.stderr:Error EINVAL: invalid command 2024-09-22T03:10:02.718 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-22T03:10:02.719 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-22T03:10:02.719 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-09-22T03:10:02.857 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:10:02.850+0000 7f52c7997640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:10:02.862 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:10:02.860+0000 7f52c7997640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:10:02.932 INFO:teuthology.orchestra.run.smithi153.stderr:no valid command found; 10 closest matches: 2024-09-22T03:10:02.932 INFO:teuthology.orchestra.run.smithi153.stderr:config get 2024-09-22T03:10:02.932 INFO:teuthology.orchestra.run.smithi153.stderr:config help 2024-09-22T03:10:02.933 INFO:teuthology.orchestra.run.smithi153.stderr:config set ... 2024-09-22T03:10:02.933 INFO:teuthology.orchestra.run.smithi153.stderr:config show 2024-09-22T03:10:02.933 INFO:teuthology.orchestra.run.smithi153.stderr:deep_scrub 2024-09-22T03:10:02.933 INFO:teuthology.orchestra.run.smithi153.stderr:dump_historic_ops 2024-09-22T03:10:02.933 INFO:teuthology.orchestra.run.smithi153.stderr:dump_historic_slow_ops 2024-09-22T03:10:02.933 INFO:teuthology.orchestra.run.smithi153.stderr:dump_metrics [] 2024-09-22T03:10:02.933 INFO:teuthology.orchestra.run.smithi153.stderr:dump_ops_in_flight 2024-09-22T03:10:02.933 INFO:teuthology.orchestra.run.smithi153.stderr:dump_pgstate_history 2024-09-22T03:10:02.933 INFO:teuthology.orchestra.run.smithi153.stderr:Error EINVAL: invalid command 2024-09-22T03:10:02.937 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-22T03:10:02.938 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-22T03:10:02.938 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-09-22T03:10:03.072 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:10:03.062+0000 7fd94684f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:10:03.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:10:03.078+0000 7fd94684f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:10:03.106 INFO:teuthology.orchestra.run.smithi153.stderr:no valid command found; 10 closest matches: 2024-09-22T03:10:03.106 INFO:teuthology.orchestra.run.smithi153.stderr:config get 2024-09-22T03:10:03.106 INFO:teuthology.orchestra.run.smithi153.stderr:config help 2024-09-22T03:10:03.106 INFO:teuthology.orchestra.run.smithi153.stderr:config set ... 2024-09-22T03:10:03.106 INFO:teuthology.orchestra.run.smithi153.stderr:config show 2024-09-22T03:10:03.106 INFO:teuthology.orchestra.run.smithi153.stderr:deep_scrub 2024-09-22T03:10:03.106 INFO:teuthology.orchestra.run.smithi153.stderr:dump_historic_ops 2024-09-22T03:10:03.106 INFO:teuthology.orchestra.run.smithi153.stderr:dump_historic_slow_ops 2024-09-22T03:10:03.106 INFO:teuthology.orchestra.run.smithi153.stderr:dump_metrics [] 2024-09-22T03:10:03.107 INFO:teuthology.orchestra.run.smithi153.stderr:dump_ops_in_flight 2024-09-22T03:10:03.107 INFO:teuthology.orchestra.run.smithi153.stderr:dump_pgstate_history 2024-09-22T03:10:03.107 INFO:teuthology.orchestra.run.smithi153.stderr:Error EINVAL: invalid command 2024-09-22T03:10:03.110 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-22T03:10:03.110 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-22T03:10:03.110 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-09-22T03:10:03.258 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:10:03.251+0000 7f5fc566c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:10:03.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:10:03.260+0000 7f5fc566c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:10:03.277 INFO:teuthology.orchestra.run.smithi153.stderr:no valid command found; 10 closest matches: 2024-09-22T03:10:03.277 INFO:teuthology.orchestra.run.smithi153.stderr:config get 2024-09-22T03:10:03.277 INFO:teuthology.orchestra.run.smithi153.stderr:config help 2024-09-22T03:10:03.277 INFO:teuthology.orchestra.run.smithi153.stderr:config set ... 2024-09-22T03:10:03.277 INFO:teuthology.orchestra.run.smithi153.stderr:config show 2024-09-22T03:10:03.277 INFO:teuthology.orchestra.run.smithi153.stderr:deep_scrub 2024-09-22T03:10:03.277 INFO:teuthology.orchestra.run.smithi153.stderr:dump_historic_ops 2024-09-22T03:10:03.278 INFO:teuthology.orchestra.run.smithi153.stderr:dump_historic_slow_ops 2024-09-22T03:10:03.278 INFO:teuthology.orchestra.run.smithi153.stderr:dump_metrics [] 2024-09-22T03:10:03.278 INFO:teuthology.orchestra.run.smithi153.stderr:dump_ops_in_flight 2024-09-22T03:10:03.278 INFO:teuthology.orchestra.run.smithi153.stderr:dump_pgstate_history 2024-09-22T03:10:03.278 INFO:teuthology.orchestra.run.smithi153.stderr:Error EINVAL: invalid command 2024-09-22T03:10:03.281 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-22T03:10:03.281 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-22T03:10:03.295 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 5 16 1905 1889 2.95134 3.03125 0.0436008 0.0420972 2024-09-22T03:10:04.085 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-22T03:10:04.244 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:10:04.240+0000 7f174cb97640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:10:04.248 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:10:04.246+0000 7f174cb97640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:10:04.295 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 6 16 2268 2252 2.93207 2.83594 0.0167294 0.0425072 2024-09-22T03:10:05.296 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 7 16 2668 2652 2.9596 3.125 0.0404665 0.0420293 2024-09-22T03:10:06.296 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 8 16 3051 3035 2.96364 2.99219 0.0706805 0.0419776 2024-09-22T03:10:06.474 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-22T03:10:06.629 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:10:06.625+0000 7fcbe9d97640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:10:06.634 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:10:06.633+0000 7fcbe9d97640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:10:07.296 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 9 16 3420 3404 2.95464 2.88281 0.0237505 0.0422199 2024-09-22T03:10:08.282 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [1, 2, 3, 0] 2024-09-22T03:10:08.283 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-22T03:10:08.283 INFO:tasks.thrashosds.thrasher:Killing osd 1, live_osds are [1, 2, 3, 0] 2024-09-22T03:10:08.283 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-09-22T03:10:08.283 INFO:teuthology.orchestra.run:waiting for 300 2024-09-22T03:10:08.296 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 10 16 3781 3765 2.94118 2.82031 0.0634805 0.0423201 2024-09-22T03:10:08.374 INFO:tasks.ceph.osd.1:Stopped 2024-09-22T03:10:08.844 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-22T03:10:08.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:10:08.943+0000 7f999a797640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:10:08.949 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:10:08.948+0000 7f999a797640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:10:09.296 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 11 16 3781 3765 2.67378 0 - 0.0423201 2024-09-22T03:10:10.297 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 12 16 3781 3765 2.45095 0 - 0.0423201 2024-09-22T03:10:11.296 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 13 16 3781 3765 2.26241 0 - 0.0423201 2024-09-22T03:10:12.297 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 14 16 3781 3765 2.1008 0 - 0.0423201 2024-09-22T03:10:13.297 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 15 16 3781 3765 1.96074 0 - 0.0423201 2024-09-22T03:10:13.375 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [1] live_osds: [2, 3, 0] 2024-09-22T03:10:13.375 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-22T03:10:13.376 INFO:tasks.thrashosds.thrasher:Reviving osd 1 2024-09-22T03:10:13.376 INFO:tasks.ceph.osd.1:Restarting daemon 2024-09-22T03:10:13.376 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2>> /var/log/ceph/ceph-osd.1.log 2024-09-22T03:10:13.379 INFO:tasks.ceph.osd.1:Started 2024-09-22T03:10:13.379 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-22T03:10:13.490 INFO:teuthology.orchestra.run.smithi153.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-09-22T03:10:13.494 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-22T03:10:13.494 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2024-09-22T03:10:14.297 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 16 16 3781 3765 1.8382 0 - 0.0423201 2024-09-22T03:10:15.297 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 17 16 3781 3765 1.73006 0 - 0.0423201 2024-09-22T03:10:16.297 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 18 16 3781 3765 1.63395 0 - 0.0423201 2024-09-22T03:10:17.297 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 19 16 3781 3765 1.54795 0 - 0.0423201 2024-09-22T03:10:18.297 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:2024-09-22T03:10:18.296800+0000 min lat: 0.00878942 max lat: 0.111418 avg lat: 0.0423201 2024-09-22T03:10:18.297 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-22T03:10:18.297 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 20 16 3781 3765 1.47055 0 - 0.0423201 2024-09-22T03:10:18.496 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-22T03:10:18.592 INFO:teuthology.orchestra.run.smithi153.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-09-22T03:10:18.594 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-22T03:10:18.594 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2024-09-22T03:10:19.297 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 21 16 3781 3765 1.40053 0 - 0.0423201 2024-09-22T03:10:19.879 INFO:teuthology.orchestra.run.smithi153.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-09-22T03:10:19.976 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-22T03:10:19.977 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-22T03:10:20.297 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 22 16 3781 3765 1.33686 0 - 0.0423201 2024-09-22T03:10:21.297 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 23 16 3781 3765 1.27874 0 - 0.0423201 2024-09-22T03:10:21.978 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-22T03:10:22.126 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:10:22.122+0000 7fbede797640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:10:22.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:10:22.129+0000 7fbede797640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:10:22.298 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 24 16 3873 3857 1.2554 0.0513393 0.0512786 0.0994808 2024-09-22T03:10:23.298 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 25 16 4189 4173 1.30393 2.46875 0.0692171 0.095762 2024-09-22T03:10:23.596 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-22T03:10:23.775 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-22T03:10:23.775 INFO:teuthology.orchestra.run.smithi153.stdout: "ops_in_flight": [ 2024-09-22T03:10:23.775 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-22T03:10:23.775 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "client_request", 2024-09-22T03:10:23.775 INFO:teuthology.orchestra.run.smithi153.stdout: "id": 485, 2024-09-22T03:10:23.775 INFO:teuthology.orchestra.run.smithi153.stdout: "detail": { 2024-09-22T03:10:23.775 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.775 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:10:23.776 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.142671+0000" 2024-09-22T03:10:23.776 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.776 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.776 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:10:23.776 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.142677+0000" 2024-09-22T03:10:23.776 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.776 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.776 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.776 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.776 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.776 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.776 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:10:23.776 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.143010+0000" 2024-09-22T03:10:23.776 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.777 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.777 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.777 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.777 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.777 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.777 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:10:23.777 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.777 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.777 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.777 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.777 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.777 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.777 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.778 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:10:23.778 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.145483+0000" 2024-09-22T03:10:23.778 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.778 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.778 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.778 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.778 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.778 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.778 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:10:23.778 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.145741+0000" 2024-09-22T03:10:23.778 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.778 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.778 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.778 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.779 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.779 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.779 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:10:23.779 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.779 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.779 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.779 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.779 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.779 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.779 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.779 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:10:23.779 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.212775+0000" 2024-09-22T03:10:23.779 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.779 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.779 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:10:23.780 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.146256+0000" 2024-09-22T03:10:23.780 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.780 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.780 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.780 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.780 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.780 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.780 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:10:23.780 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.780 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.780 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.780 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.780 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.780 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.781 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.781 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:10:23.781 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.146840+0000" 2024-09-22T03:10:23.781 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.781 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.781 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.781 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.781 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.781 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.781 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:10:23.781 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.781 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.781 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.781 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.782 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.782 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.782 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.782 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:10:23.782 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.147224+0000" 2024-09-22T03:10:23.782 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.782 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.782 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.782 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.782 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.782 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.782 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:10:23.782 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.782 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.783 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.783 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.783 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.783 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.783 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.783 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:10:23.783 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.783 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.783 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.783 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.783 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.783 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.783 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.783 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:10:23.784 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.784 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.784 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.784 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.784 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.784 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.784 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.784 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:10:23.784 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.148308+0000" 2024-09-22T03:10:23.784 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.784 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.784 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.784 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.785 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.785 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.785 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:10:23.785 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.785 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.785 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.785 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.785 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.785 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.785 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.785 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:10:23.785 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.149060+0000" 2024-09-22T03:10:23.785 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.785 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.786 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.786 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.786 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.786 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.786 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:10:23.786 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.786 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.786 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.786 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.786 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.786 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.786 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.787 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:10:23.787 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.210419+0000" 2024-09-22T03:10:23.787 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.787 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.787 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.787 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.787 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.787 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.787 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:10:23.787 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.787 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:10:23.787 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:10:23.787 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.787 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-22T03:10:23.788 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "client_request", 2024-09-22T03:10:23.788 INFO:teuthology.orchestra.run.smithi153.stdout: "id": 487, 2024-09-22T03:10:23.788 INFO:teuthology.orchestra.run.smithi153.stdout: "detail": { 2024-09-22T03:10:23.788 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.788 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:10:23.788 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.151647+0000" 2024-09-22T03:10:23.788 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.788 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.788 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:10:23.788 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.151655+0000" 2024-09-22T03:10:23.788 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.788 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.788 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.788 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.788 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.789 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.789 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:10:23.789 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.151980+0000" 2024-09-22T03:10:23.789 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.789 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.789 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.789 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.789 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.789 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.789 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:10:23.789 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.789 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.789 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.789 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.790 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.790 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.790 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.790 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:10:23.790 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.152481+0000" 2024-09-22T03:10:23.790 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.790 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.790 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.790 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.790 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.790 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.790 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:10:23.790 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.154464+0000" 2024-09-22T03:10:23.790 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.791 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.791 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.791 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.791 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.791 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.791 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:10:23.791 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.791 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.791 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.791 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.791 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.791 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.791 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.791 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:10:23.791 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.225454+0000" 2024-09-22T03:10:23.792 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.792 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.792 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:10:23.792 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.155397+0000" 2024-09-22T03:10:23.792 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.792 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.792 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.792 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.792 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.792 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.792 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:10:23.792 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.792 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.792 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.793 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.793 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.793 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.793 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.793 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:10:23.793 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.155722+0000" 2024-09-22T03:10:23.793 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.793 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.793 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.793 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.793 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.793 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.793 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:10:23.793 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.793 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.794 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.794 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.794 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.794 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.794 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.794 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:10:23.794 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.156563+0000" 2024-09-22T03:10:23.794 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.794 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.794 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.794 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.794 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.794 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.794 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:10:23.795 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.795 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.795 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.795 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.795 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.795 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.795 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.795 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:10:23.795 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.795 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.795 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.795 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.795 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.795 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.796 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.796 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:10:23.796 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.796 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.796 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.796 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.796 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.796 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.796 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.796 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:10:23.796 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.157383+0000" 2024-09-22T03:10:23.796 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.796 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.796 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.797 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.797 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.797 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.797 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:10:23.797 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.797 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.797 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.797 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.797 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.797 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.797 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.797 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:10:23.797 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.158261+0000" 2024-09-22T03:10:23.797 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.798 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.798 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.798 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.798 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.798 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.798 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:10:23.798 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.798 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.798 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.798 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.798 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.798 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.798 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.799 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:10:23.799 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.223734+0000" 2024-09-22T03:10:23.799 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.799 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.799 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.799 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.799 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.799 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.799 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:10:23.799 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.799 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:10:23.799 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:10:23.799 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.799 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-22T03:10:23.800 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "client_request", 2024-09-22T03:10:23.800 INFO:teuthology.orchestra.run.smithi153.stdout: "id": 489, 2024-09-22T03:10:23.800 INFO:teuthology.orchestra.run.smithi153.stdout: "detail": { 2024-09-22T03:10:23.800 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.800 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:10:23.800 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.158586+0000" 2024-09-22T03:10:23.800 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.800 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.800 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:10:23.800 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.158591+0000" 2024-09-22T03:10:23.800 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.800 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.800 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.800 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.800 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.801 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.801 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:10:23.801 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.158784+0000" 2024-09-22T03:10:23.801 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.801 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.801 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.801 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.801 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.801 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.801 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:10:23.801 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.801 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.801 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.801 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.802 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.802 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.802 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.802 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:10:23.802 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.160518+0000" 2024-09-22T03:10:23.802 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.802 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.802 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.802 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.802 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.802 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.802 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:10:23.802 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.160744+0000" 2024-09-22T03:10:23.802 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.803 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.803 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.803 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.803 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.803 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.803 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:10:23.803 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.803 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.803 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.803 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.803 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.803 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.803 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.803 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:10:23.804 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.247624+0000" 2024-09-22T03:10:23.804 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.804 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.804 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:10:23.804 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.161399+0000" 2024-09-22T03:10:23.804 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.804 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.804 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.804 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.804 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.804 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.804 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:10:23.804 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.805 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.805 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.805 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.805 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.805 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.805 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.805 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:10:23.805 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.162257+0000" 2024-09-22T03:10:23.805 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.805 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.805 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.805 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.805 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.805 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.805 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:10:23.806 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.806 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.806 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.806 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.806 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.806 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.806 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.806 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:10:23.806 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.162714+0000" 2024-09-22T03:10:23.806 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.806 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.806 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.806 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.806 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.807 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.807 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:10:23.807 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.807 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.807 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.807 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.807 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.807 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.807 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.807 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:10:23.807 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.807 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.807 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.807 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.807 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.808 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.808 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.808 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:10:23.808 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.808 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.808 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.808 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.808 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.808 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.808 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.808 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:10:23.808 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.164432+0000" 2024-09-22T03:10:23.808 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.808 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.808 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.809 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.809 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.809 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.809 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:10:23.809 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.809 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.809 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.809 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.809 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.809 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.809 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.809 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:10:23.809 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.165863+0000" 2024-09-22T03:10:23.810 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.810 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.810 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.810 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.810 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.810 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.810 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:10:23.810 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.810 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.810 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.810 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.810 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.810 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.810 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.810 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:10:23.811 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.245489+0000" 2024-09-22T03:10:23.811 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.811 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.811 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.811 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.811 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.811 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.811 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:10:23.811 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.811 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:10:23.811 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:10:23.811 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.811 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-22T03:10:23.811 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "client_request", 2024-09-22T03:10:23.812 INFO:teuthology.orchestra.run.smithi153.stdout: "id": 494, 2024-09-22T03:10:23.812 INFO:teuthology.orchestra.run.smithi153.stdout: "detail": { 2024-09-22T03:10:23.812 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.812 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:10:23.812 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.185993+0000" 2024-09-22T03:10:23.812 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.812 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.812 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:10:23.812 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.186000+0000" 2024-09-22T03:10:23.812 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.812 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.812 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.812 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.812 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.812 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.813 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:10:23.813 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.186104+0000" 2024-09-22T03:10:23.813 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.813 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.813 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.813 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.813 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.813 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.813 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:10:23.813 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.813 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.813 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.813 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.813 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.813 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.814 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.814 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:10:23.814 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.186194+0000" 2024-09-22T03:10:23.814 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.814 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.814 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.814 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.814 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.814 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.814 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:10:23.814 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.186260+0000" 2024-09-22T03:10:23.814 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.814 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.814 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.815 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.815 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.815 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.815 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:10:23.815 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.815 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.815 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.815 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.815 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.815 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.815 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.815 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:10:23.815 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.257975+0000" 2024-09-22T03:10:23.815 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.815 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.816 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:10:23.816 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.186514+0000" 2024-09-22T03:10:23.816 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.816 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.816 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.816 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.816 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.816 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.816 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:10:23.816 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.816 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.816 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.816 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.816 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.817 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.817 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.817 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:10:23.817 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.186679+0000" 2024-09-22T03:10:23.817 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.817 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.817 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.817 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.817 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.817 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.817 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:10:23.817 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.817 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.817 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.817 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.818 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.818 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.818 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.818 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:10:23.818 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.186857+0000" 2024-09-22T03:10:23.818 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.818 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.818 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.818 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.818 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.818 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.818 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:10:23.818 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.818 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.818 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.819 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.819 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.819 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.819 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.819 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:10:23.819 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.819 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.819 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.819 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.819 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.819 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.819 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.819 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:10:23.820 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.820 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.820 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.820 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.820 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.820 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.820 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.820 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:10:23.820 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.187232+0000" 2024-09-22T03:10:23.820 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.820 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.820 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.820 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.820 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.821 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.821 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:10:23.821 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.821 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.821 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.821 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.821 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.821 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.821 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.821 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:10:23.821 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.189798+0000" 2024-09-22T03:10:23.821 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.821 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.821 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.821 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.822 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.822 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.822 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:10:23.822 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.822 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.822 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.822 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.822 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.822 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.822 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.822 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:10:23.822 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.257060+0000" 2024-09-22T03:10:23.822 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.822 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.823 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.823 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.823 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.823 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.823 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:10:23.823 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.823 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:10:23.823 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:10:23.823 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.823 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-22T03:10:23.823 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "client_request", 2024-09-22T03:10:23.823 INFO:teuthology.orchestra.run.smithi153.stdout: "id": 496, 2024-09-22T03:10:23.823 INFO:teuthology.orchestra.run.smithi153.stdout: "detail": { 2024-09-22T03:10:23.823 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.823 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:10:23.824 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.194971+0000" 2024-09-22T03:10:23.824 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.824 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.824 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:10:23.824 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.194986+0000" 2024-09-22T03:10:23.824 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.824 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.824 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.824 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.824 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.824 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.824 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:10:23.824 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.195871+0000" 2024-09-22T03:10:23.824 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.825 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.825 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.825 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.825 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.825 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.825 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:10:23.825 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.825 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.825 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.825 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.825 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.825 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.825 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.825 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:10:23.825 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.196075+0000" 2024-09-22T03:10:23.826 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.826 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.826 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.826 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.826 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.826 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.826 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:10:23.826 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.196558+0000" 2024-09-22T03:10:23.826 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.826 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.826 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.826 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.826 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.826 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.827 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:10:23.827 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.827 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.827 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.827 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.827 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.827 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.827 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.827 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:10:23.827 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.259946+0000" 2024-09-22T03:10:23.827 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.827 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.827 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:10:23.827 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.196864+0000" 2024-09-22T03:10:23.827 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.828 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.828 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.828 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.828 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.828 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.828 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:10:23.828 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.828 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.828 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.828 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.828 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.828 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.828 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.828 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:10:23.829 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.197012+0000" 2024-09-22T03:10:23.829 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.829 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.829 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.829 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.829 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.829 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.829 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:10:23.829 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.829 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.829 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.829 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.829 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.829 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.829 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.830 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:10:23.830 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.197188+0000" 2024-09-22T03:10:23.830 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.830 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.830 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.830 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.830 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.830 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.830 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:10:23.830 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.830 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.830 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.830 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.831 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.831 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.831 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.831 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:10:23.831 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.831 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.831 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.831 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.831 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.831 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.831 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.831 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:10:23.831 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.831 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.832 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.832 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.832 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.832 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.832 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.832 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:10:23.832 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.197432+0000" 2024-09-22T03:10:23.832 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.832 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.832 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.832 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.832 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.832 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.832 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:10:23.832 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.833 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.833 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.833 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.833 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.833 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.833 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.833 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:10:23.833 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.197979+0000" 2024-09-22T03:10:23.833 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.833 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.833 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.833 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.833 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.833 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.834 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:10:23.834 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.834 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.834 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.834 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.834 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.834 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.834 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.834 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:10:23.834 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.259391+0000" 2024-09-22T03:10:23.834 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.834 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.834 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.834 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.835 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.835 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.835 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:10:23.835 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.835 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:10:23.835 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:10:23.835 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.835 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-22T03:10:23.835 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "client_request", 2024-09-22T03:10:23.835 INFO:teuthology.orchestra.run.smithi153.stdout: "id": 497, 2024-09-22T03:10:23.835 INFO:teuthology.orchestra.run.smithi153.stdout: "detail": { 2024-09-22T03:10:23.835 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.835 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:10:23.835 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.200518+0000" 2024-09-22T03:10:23.835 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.836 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.836 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:10:23.836 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.200530+0000" 2024-09-22T03:10:23.836 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.836 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.836 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.836 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.836 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.836 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.836 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:10:23.836 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.201130+0000" 2024-09-22T03:10:23.836 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.836 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.836 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.837 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.837 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.837 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.837 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:10:23.837 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.837 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.837 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.837 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.837 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.837 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.837 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.837 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:10:23.837 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.201292+0000" 2024-09-22T03:10:23.837 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.837 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.838 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.838 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.838 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.838 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.838 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:10:23.838 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.202035+0000" 2024-09-22T03:10:23.838 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.838 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.838 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.838 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.838 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.838 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.838 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:10:23.838 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.839 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.839 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.839 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.839 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.839 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.839 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.839 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:10:23.839 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.260936+0000" 2024-09-22T03:10:23.839 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.839 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.839 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:10:23.839 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.202776+0000" 2024-09-22T03:10:23.839 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.839 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.839 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.840 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.840 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.840 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.840 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:10:23.840 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.840 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.840 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.840 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.840 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.840 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.840 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.840 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:10:23.840 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.204165+0000" 2024-09-22T03:10:23.840 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.841 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.841 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.841 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.841 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.841 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.841 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:10:23.841 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.841 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.841 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.841 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.841 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.841 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.841 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.841 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:10:23.841 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.204429+0000" 2024-09-22T03:10:23.842 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.842 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.842 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.842 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.842 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.842 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.842 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:10:23.842 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.842 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.842 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.842 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.842 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.842 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.843 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.843 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:10:23.843 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.843 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.843 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.843 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.843 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.843 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.843 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.843 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:10:23.843 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.843 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.843 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.843 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.844 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.844 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.844 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.844 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:10:23.844 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.205282+0000" 2024-09-22T03:10:23.844 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.844 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.844 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.844 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.844 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.844 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.844 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:10:23.844 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.844 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.844 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.845 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.845 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.845 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.845 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.845 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:10:23.845 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.205866+0000" 2024-09-22T03:10:23.845 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.845 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.845 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.845 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.845 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.845 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.845 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:10:23.845 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.845 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.846 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.846 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.846 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.846 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.846 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.846 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:10:23.846 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.260581+0000" 2024-09-22T03:10:23.846 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.846 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.846 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.846 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.846 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.846 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.846 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:10:23.847 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.847 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:10:23.847 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:10:23.847 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.847 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-22T03:10:23.847 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "client_request", 2024-09-22T03:10:23.847 INFO:teuthology.orchestra.run.smithi153.stdout: "id": 499, 2024-09-22T03:10:23.847 INFO:teuthology.orchestra.run.smithi153.stdout: "detail": { 2024-09-22T03:10:23.847 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.847 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:10:23.847 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.204869+0000" 2024-09-22T03:10:23.847 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.847 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.847 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:10:23.847 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.204884+0000" 2024-09-22T03:10:23.848 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.848 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.848 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.848 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.848 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.848 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.848 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:10:23.848 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.205355+0000" 2024-09-22T03:10:23.848 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.848 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.848 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.848 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.848 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.848 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.848 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:10:23.849 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.849 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.849 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.849 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.849 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.849 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.849 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.849 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:10:23.849 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.205958+0000" 2024-09-22T03:10:23.849 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.849 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.849 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.849 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.849 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.850 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.850 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:10:23.850 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.206286+0000" 2024-09-22T03:10:23.850 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.850 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.850 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.850 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.850 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.850 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.850 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:10:23.850 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.850 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.850 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.850 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.850 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.851 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.851 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.851 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:10:23.851 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.263218+0000" 2024-09-22T03:10:23.851 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.851 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.851 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:10:23.851 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.208500+0000" 2024-09-22T03:10:23.851 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.851 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.851 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.851 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.851 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.851 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.851 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:10:23.852 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.852 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.852 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.852 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.852 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.852 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.852 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.852 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:10:23.852 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.209526+0000" 2024-09-22T03:10:23.852 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.852 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.852 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.852 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.853 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.853 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.853 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:10:23.853 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.853 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.853 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.853 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.853 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.853 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.853 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.853 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:10:23.853 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.210077+0000" 2024-09-22T03:10:23.853 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.853 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.854 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.854 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.854 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.854 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.854 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:10:23.854 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.854 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.854 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.854 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.854 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.854 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.854 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.854 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:10:23.854 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.855 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.855 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.855 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.855 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.855 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.855 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.855 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:10:23.855 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.855 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.855 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.855 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.855 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.855 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.855 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.856 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:10:23.856 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.210591+0000" 2024-09-22T03:10:23.856 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.856 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.856 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.856 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.856 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.856 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.856 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:10:23.856 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.856 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.856 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.856 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.856 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.857 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.857 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.857 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:10:23.857 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.211645+0000" 2024-09-22T03:10:23.857 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.857 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.857 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.857 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.857 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.857 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.857 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:10:23.857 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.857 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.857 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.857 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.858 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.858 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.858 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.858 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:10:23.858 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.262800+0000" 2024-09-22T03:10:23.858 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.858 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.858 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.858 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.858 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.858 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.858 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:10:23.858 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.858 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:10:23.859 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:10:23.859 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.859 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-22T03:10:23.859 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "client_request", 2024-09-22T03:10:23.859 INFO:teuthology.orchestra.run.smithi153.stdout: "id": 500, 2024-09-22T03:10:23.859 INFO:teuthology.orchestra.run.smithi153.stdout: "detail": { 2024-09-22T03:10:23.859 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.859 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:10:23.859 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.214990+0000" 2024-09-22T03:10:23.859 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.859 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.859 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:10:23.859 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.214996+0000" 2024-09-22T03:10:23.859 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.859 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.860 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.860 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.860 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.860 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.860 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:10:23.860 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.215123+0000" 2024-09-22T03:10:23.860 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.860 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.860 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.860 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.860 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.860 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.860 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:10:23.860 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.860 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.861 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.861 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.861 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.861 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.861 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.861 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:10:23.861 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.215205+0000" 2024-09-22T03:10:23.861 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.861 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.861 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.861 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.861 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.861 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.861 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:10:23.862 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.215689+0000" 2024-09-22T03:10:23.862 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.862 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.862 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.862 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.862 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.862 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.862 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:10:23.862 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.862 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.862 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.862 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.863 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.863 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.863 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.863 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:10:23.863 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.271716+0000" 2024-09-22T03:10:23.863 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.863 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.863 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:10:23.863 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.216773+0000" 2024-09-22T03:10:23.863 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.863 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.863 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.863 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.863 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.863 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.864 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:10:23.864 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.864 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.864 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.864 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.864 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.864 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.864 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.864 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:10:23.864 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.217126+0000" 2024-09-22T03:10:23.864 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.864 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.864 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.865 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.865 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.865 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.865 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:10:23.865 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.865 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.865 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.865 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.865 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.865 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.865 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.865 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:10:23.865 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.217327+0000" 2024-09-22T03:10:23.865 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.865 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.866 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.866 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.866 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.866 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.866 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:10:23.866 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.866 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.866 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.866 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.866 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.866 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.866 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.866 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:10:23.867 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.867 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.867 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.867 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.867 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.867 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.867 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.867 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:10:23.867 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.867 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.867 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.867 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.867 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.867 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.867 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.868 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:10:23.868 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.218134+0000" 2024-09-22T03:10:23.868 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.868 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.868 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.868 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.868 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.868 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.868 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:10:23.868 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.868 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.868 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.868 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.868 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.869 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.869 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.869 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:10:23.869 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.220083+0000" 2024-09-22T03:10:23.869 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.869 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.869 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.869 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.869 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.869 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.869 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:10:23.869 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.869 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.869 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.869 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.870 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.870 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.870 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.870 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:10:23.870 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.269483+0000" 2024-09-22T03:10:23.870 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.870 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.870 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.870 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.870 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.870 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.870 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:10:23.870 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.870 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:10:23.870 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:10:23.871 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.871 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-22T03:10:23.871 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "client_request", 2024-09-22T03:10:23.871 INFO:teuthology.orchestra.run.smithi153.stdout: "id": 502, 2024-09-22T03:10:23.871 INFO:teuthology.orchestra.run.smithi153.stdout: "detail": { 2024-09-22T03:10:23.871 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.871 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:10:23.871 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.221782+0000" 2024-09-22T03:10:23.871 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.871 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.871 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:10:23.871 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.221794+0000" 2024-09-22T03:10:23.871 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.871 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.872 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.872 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.872 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.872 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.872 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:10:23.872 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.224116+0000" 2024-09-22T03:10:23.872 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.872 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.872 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.872 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.872 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.872 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.872 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:10:23.872 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.872 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.873 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.873 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.873 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.873 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.873 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.873 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:10:23.873 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.225001+0000" 2024-09-22T03:10:23.873 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.873 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.873 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.873 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.873 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.873 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.873 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:10:23.873 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.225710+0000" 2024-09-22T03:10:23.874 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.874 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.874 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.874 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.874 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.874 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.874 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:10:23.874 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.874 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.874 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.874 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.874 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.874 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.874 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.874 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:10:23.875 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.275490+0000" 2024-09-22T03:10:23.875 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.875 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.875 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:10:23.875 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.226842+0000" 2024-09-22T03:10:23.875 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.875 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.875 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.875 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.875 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.875 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.875 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:10:23.875 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.876 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.876 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.876 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.876 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.876 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.876 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.876 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:10:23.876 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.226987+0000" 2024-09-22T03:10:23.876 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.876 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.876 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.876 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.876 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.876 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.876 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:10:23.877 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.877 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.877 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.877 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.877 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.877 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.877 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.877 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:10:23.877 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.227715+0000" 2024-09-22T03:10:23.877 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.877 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.877 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.877 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.877 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.878 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.878 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:10:23.878 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.878 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.878 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.878 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.878 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.878 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.878 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.878 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:10:23.878 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.878 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.878 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.878 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.879 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.879 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.879 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.879 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:10:23.879 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.879 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.879 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.879 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.879 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.879 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.879 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.879 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:10:23.879 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.228070+0000" 2024-09-22T03:10:23.879 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.879 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.880 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.880 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.880 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.880 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.880 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:10:23.880 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.880 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.880 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.880 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.880 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.880 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.880 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.880 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:10:23.880 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.228842+0000" 2024-09-22T03:10:23.880 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.881 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.881 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.881 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.881 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.881 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.881 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:10:23.881 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.881 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.881 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.881 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.881 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.881 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.881 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.881 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:10:23.881 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.274874+0000" 2024-09-22T03:10:23.882 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.882 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.882 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.882 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.882 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.882 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.882 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:10:23.882 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.882 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:10:23.882 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:10:23.882 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.882 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-22T03:10:23.882 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "client_request", 2024-09-22T03:10:23.882 INFO:teuthology.orchestra.run.smithi153.stdout: "id": 503, 2024-09-22T03:10:23.883 INFO:teuthology.orchestra.run.smithi153.stdout: "detail": { 2024-09-22T03:10:23.883 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.883 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:10:23.883 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.227607+0000" 2024-09-22T03:10:23.883 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.883 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.883 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:10:23.883 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.227613+0000" 2024-09-22T03:10:23.883 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.883 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.883 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.883 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.883 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.883 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.883 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:10:23.884 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.227991+0000" 2024-09-22T03:10:23.884 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.884 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.884 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.884 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.884 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.884 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.884 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:10:23.884 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.884 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.884 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.884 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.884 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.884 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.885 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.885 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:10:23.885 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.228500+0000" 2024-09-22T03:10:23.885 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.885 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.885 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.885 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.885 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.885 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.885 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:10:23.885 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.228823+0000" 2024-09-22T03:10:23.885 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.885 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.885 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.885 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.886 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.886 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.886 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:10:23.886 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.886 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.886 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.886 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.886 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.886 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.886 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.886 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:10:23.886 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.277979+0000" 2024-09-22T03:10:23.886 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.887 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.887 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:10:23.887 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.229174+0000" 2024-09-22T03:10:23.887 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.887 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.887 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.887 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.887 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.887 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.887 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:10:23.887 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.887 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.887 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.887 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.887 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.888 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.888 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.888 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:10:23.888 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.229549+0000" 2024-09-22T03:10:23.888 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.888 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.888 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.888 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.888 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.888 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.888 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:10:23.888 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.888 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.888 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.889 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.889 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.889 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.889 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.889 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:10:23.889 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.231675+0000" 2024-09-22T03:10:23.889 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.889 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.889 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.889 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.889 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.889 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.889 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:10:23.889 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.889 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.890 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.890 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.890 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.890 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.890 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.890 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:10:23.890 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.890 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.890 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.890 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.890 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.890 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.890 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.890 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:10:23.890 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.891 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.891 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.891 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.891 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.891 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.891 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.891 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:10:23.891 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.232107+0000" 2024-09-22T03:10:23.891 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.891 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.891 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.891 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.891 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.891 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.892 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:10:23.892 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.892 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.892 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.892 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.892 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.892 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.892 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.892 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:10:23.892 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.233286+0000" 2024-09-22T03:10:23.892 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.892 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.892 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.892 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.892 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.893 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.893 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:10:23.893 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.893 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.893 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.893 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.893 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.893 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.893 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.893 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:10:23.893 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.277029+0000" 2024-09-22T03:10:23.893 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.893 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.893 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.893 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.894 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.894 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.894 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:10:23.894 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.894 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:10:23.894 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:10:23.894 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.894 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-22T03:10:23.894 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "client_request", 2024-09-22T03:10:23.894 INFO:teuthology.orchestra.run.smithi153.stdout: "id": 505, 2024-09-22T03:10:23.894 INFO:teuthology.orchestra.run.smithi153.stdout: "detail": { 2024-09-22T03:10:23.894 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.894 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:10:23.894 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.239395+0000" 2024-09-22T03:10:23.895 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.895 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.895 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:10:23.895 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.239412+0000" 2024-09-22T03:10:23.895 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.895 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.895 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.895 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.895 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.895 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.895 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:10:23.895 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.239968+0000" 2024-09-22T03:10:23.895 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.895 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.895 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.896 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.896 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.896 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.896 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:10:23.896 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.896 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.896 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.896 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.896 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.896 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.896 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.896 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:10:23.896 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.240121+0000" 2024-09-22T03:10:23.896 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.897 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.897 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.897 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.897 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.897 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.897 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:10:23.897 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.240923+0000" 2024-09-22T03:10:23.897 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.897 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.897 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.897 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.897 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.897 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.898 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:10:23.898 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.898 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.898 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.898 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.898 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.898 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.898 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.898 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:10:23.898 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.284302+0000" 2024-09-22T03:10:23.898 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.898 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.898 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:10:23.898 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.241230+0000" 2024-09-22T03:10:23.899 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.899 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.899 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.899 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.899 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.899 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.899 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:10:23.899 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.899 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.899 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.899 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.899 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.899 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.899 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.899 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:10:23.900 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.241982+0000" 2024-09-22T03:10:23.900 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.900 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.900 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.900 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.900 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.900 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.900 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:10:23.900 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.900 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.900 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.900 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.900 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.901 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.901 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.901 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:10:23.901 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.242516+0000" 2024-09-22T03:10:23.901 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.901 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.901 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.901 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.901 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.901 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.901 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:10:23.901 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.901 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.901 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.902 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.902 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.902 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.902 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.902 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:10:23.902 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.902 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.902 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.902 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.902 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.902 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.902 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.902 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:10:23.902 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.902 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.903 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.903 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.903 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.903 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.903 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.903 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:10:23.903 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.243871+0000" 2024-09-22T03:10:23.903 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.903 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.903 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.903 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.903 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.903 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.903 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:10:23.904 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.904 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.904 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.904 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.904 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.904 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.904 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.904 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:10:23.904 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.244475+0000" 2024-09-22T03:10:23.904 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.904 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.904 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.904 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.904 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.904 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.905 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:10:23.905 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.905 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.905 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.905 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.905 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.905 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.905 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.905 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:10:23.905 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.283695+0000" 2024-09-22T03:10:23.905 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.905 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.905 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.905 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.905 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.906 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.906 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:10:23.906 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.906 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:10:23.906 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:10:23.906 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.906 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-22T03:10:23.906 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "client_request", 2024-09-22T03:10:23.906 INFO:teuthology.orchestra.run.smithi153.stdout: "id": 575, 2024-09-22T03:10:23.906 INFO:teuthology.orchestra.run.smithi153.stdout: "detail": { 2024-09-22T03:10:23.906 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.906 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:10:23.906 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.493687+0000" 2024-09-22T03:10:23.906 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.907 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.907 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:10:23.907 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.493698+0000" 2024-09-22T03:10:23.907 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.907 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.907 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.907 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.907 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.907 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.907 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:10:23.907 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.496042+0000" 2024-09-22T03:10:23.907 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.907 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.907 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.907 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.908 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.908 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.908 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:10:23.908 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.908 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.908 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.908 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.908 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.908 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.908 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.908 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:10:23.908 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.496264+0000" 2024-09-22T03:10:23.908 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.909 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.909 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.909 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.909 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.909 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.909 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:10:23.909 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.496781+0000" 2024-09-22T03:10:23.909 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.909 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.909 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.909 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.909 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.909 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.909 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:10:23.909 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.910 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.910 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.910 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.910 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.910 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.910 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.910 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:10:23.910 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.568352+0000" 2024-09-22T03:10:23.910 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.910 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.910 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:10:23.910 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.497220+0000" 2024-09-22T03:10:23.910 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.910 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.911 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.911 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.911 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.911 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.911 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:10:23.911 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.911 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.911 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.911 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.911 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.911 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.911 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.911 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:10:23.911 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.497432+0000" 2024-09-22T03:10:23.911 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.912 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.912 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.912 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.912 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.912 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.912 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:10:23.912 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.912 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.912 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.912 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.912 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.912 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.912 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.912 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:10:23.913 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.499146+0000" 2024-09-22T03:10:23.913 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.913 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.913 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.913 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.913 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.913 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.913 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:10:23.913 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.913 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.913 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.913 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.913 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.913 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.913 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.914 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:10:23.914 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.914 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.914 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.914 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.914 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.914 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.914 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.914 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:10:23.914 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.914 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.914 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.914 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.914 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.915 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.915 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.915 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:10:23.915 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.499492+0000" 2024-09-22T03:10:23.915 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.915 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.915 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.915 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.915 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.915 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.915 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:10:23.915 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.915 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.915 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.915 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.916 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.916 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.916 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.916 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:10:23.916 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.503512+0000" 2024-09-22T03:10:23.916 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.916 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.916 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.916 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.916 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.916 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.916 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:10:23.916 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.916 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.916 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.917 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.917 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.917 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.917 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.917 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:10:23.917 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.567521+0000" 2024-09-22T03:10:23.917 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.917 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.917 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.917 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.917 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.917 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.917 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:10:23.917 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.917 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:10:23.918 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:10:23.918 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.918 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-22T03:10:23.918 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "client_request", 2024-09-22T03:10:23.918 INFO:teuthology.orchestra.run.smithi153.stdout: "id": 577, 2024-09-22T03:10:23.918 INFO:teuthology.orchestra.run.smithi153.stdout: "detail": { 2024-09-22T03:10:23.918 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.918 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:10:23.918 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.504128+0000" 2024-09-22T03:10:23.918 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.918 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.918 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:10:23.918 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.504140+0000" 2024-09-22T03:10:23.918 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.918 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.919 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.919 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.919 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.919 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.919 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:10:23.919 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.504943+0000" 2024-09-22T03:10:23.919 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.919 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.919 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.919 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.919 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.919 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.919 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:10:23.920 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.920 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.920 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.920 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.920 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.920 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.920 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.920 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:10:23.920 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.506261+0000" 2024-09-22T03:10:23.920 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.920 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.920 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.920 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.920 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.921 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.921 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:10:23.921 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.508015+0000" 2024-09-22T03:10:23.921 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.921 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.921 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.921 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.921 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.921 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.921 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:10:23.921 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.921 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.921 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.921 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.921 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.922 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.922 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.922 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:10:23.922 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.570029+0000" 2024-09-22T03:10:23.922 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.922 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.922 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:10:23.922 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.508379+0000" 2024-09-22T03:10:23.922 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.922 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.922 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.922 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.922 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.922 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.923 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:10:23.923 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.923 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.923 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.923 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.923 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.923 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.923 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.923 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:10:23.923 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.508698+0000" 2024-09-22T03:10:23.923 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.923 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.923 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.923 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.924 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.924 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.924 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:10:23.924 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.924 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.924 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.924 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.924 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.924 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.924 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.924 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:10:23.924 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.508835+0000" 2024-09-22T03:10:23.924 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.924 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.924 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.925 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.925 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.925 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.925 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:10:23.925 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.925 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.925 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.925 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.925 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.925 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.925 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.925 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:10:23.925 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.925 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.925 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.926 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.926 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.926 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.926 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.926 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:10:23.926 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.926 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.926 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.926 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.926 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.926 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.926 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.926 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:10:23.926 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.510026+0000" 2024-09-22T03:10:23.927 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.927 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.927 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.927 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.927 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.927 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.927 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:10:23.927 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.927 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.927 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.927 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.927 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.927 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.927 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.927 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:10:23.928 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.510590+0000" 2024-09-22T03:10:23.928 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.928 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.928 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.928 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.928 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.928 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.928 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:10:23.928 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.928 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.928 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.928 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.928 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.928 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.929 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.929 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:10:23.929 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.569362+0000" 2024-09-22T03:10:23.929 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.929 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.929 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.929 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.929 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.929 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.929 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:10:23.929 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.929 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:10:23.929 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:10:23.929 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.930 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-22T03:10:23.930 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "client_request", 2024-09-22T03:10:23.930 INFO:teuthology.orchestra.run.smithi153.stdout: "id": 579, 2024-09-22T03:10:23.930 INFO:teuthology.orchestra.run.smithi153.stdout: "detail": { 2024-09-22T03:10:23.930 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.930 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:10:23.930 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.515244+0000" 2024-09-22T03:10:23.930 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.930 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.930 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:10:23.930 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.515251+0000" 2024-09-22T03:10:23.930 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.930 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.930 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.931 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.931 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.931 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.931 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:10:23.931 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.515633+0000" 2024-09-22T03:10:23.931 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.931 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.931 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.931 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.931 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.931 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.931 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:10:23.931 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.931 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.931 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.932 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.932 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.932 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.932 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.932 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:10:23.932 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.515798+0000" 2024-09-22T03:10:23.932 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.932 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.932 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.932 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.932 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.932 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.932 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:10:23.932 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.515954+0000" 2024-09-22T03:10:23.932 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.933 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.933 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.933 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.933 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.933 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.933 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:10:23.933 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.933 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.933 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.933 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.933 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.933 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.933 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.933 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:10:23.934 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.571013+0000" 2024-09-22T03:10:23.934 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.934 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.934 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:10:23.934 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.517668+0000" 2024-09-22T03:10:23.934 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.934 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.934 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.934 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.934 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.934 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.934 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:10:23.934 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.934 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.935 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.935 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.935 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.935 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.935 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.935 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:10:23.935 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.518050+0000" 2024-09-22T03:10:23.935 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.935 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.935 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.935 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.935 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.935 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.935 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:10:23.935 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.936 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.936 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.936 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.936 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.936 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.936 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.936 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:10:23.936 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.518399+0000" 2024-09-22T03:10:23.936 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.936 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.936 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.936 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.936 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.936 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.936 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:10:23.937 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.937 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.937 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.937 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.937 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.937 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.937 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.937 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:10:23.937 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.937 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.937 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.937 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.937 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.937 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.938 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.938 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:10:23.938 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.938 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.938 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.938 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.938 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.938 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.938 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.938 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:10:23.938 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.518723+0000" 2024-09-22T03:10:23.938 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.938 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.938 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.938 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.939 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.939 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.939 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:10:23.939 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.939 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.939 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.939 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.939 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.939 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.939 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.939 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:10:23.939 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.519646+0000" 2024-09-22T03:10:23.939 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.939 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.939 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.940 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.940 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.940 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.940 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:10:23.940 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.940 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.940 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.940 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.940 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.940 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.940 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.940 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:10:23.940 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.570667+0000" 2024-09-22T03:10:23.941 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.941 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.941 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.941 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.941 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.941 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.941 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:10:23.941 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.941 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:10:23.941 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:10:23.941 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.941 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-22T03:10:23.941 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "client_request", 2024-09-22T03:10:23.941 INFO:teuthology.orchestra.run.smithi153.stdout: "id": 580, 2024-09-22T03:10:23.941 INFO:teuthology.orchestra.run.smithi153.stdout: "detail": { 2024-09-22T03:10:23.942 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.942 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:10:23.942 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.520399+0000" 2024-09-22T03:10:23.942 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.942 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.942 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:10:23.942 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.520405+0000" 2024-09-22T03:10:23.942 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.942 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.942 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.942 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.942 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.945 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.946 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:10:23.946 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.521835+0000" 2024-09-22T03:10:23.946 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.946 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.946 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.946 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.946 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.946 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.946 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:10:23.946 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.946 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.946 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.946 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.947 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.947 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.947 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.947 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:10:23.947 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.522097+0000" 2024-09-22T03:10:23.947 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.947 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.947 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.947 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.947 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.947 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.947 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:10:23.947 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.522456+0000" 2024-09-22T03:10:23.947 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.948 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.948 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.948 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.948 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.948 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.948 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:10:23.948 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.948 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.948 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.948 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.948 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.948 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.948 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.948 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:10:23.948 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.571864+0000" 2024-09-22T03:10:23.949 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.949 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.949 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:10:23.949 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.523487+0000" 2024-09-22T03:10:23.949 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.949 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.949 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.949 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.949 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.949 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.949 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:10:23.949 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.949 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.949 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.949 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.950 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.950 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.950 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.950 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:10:23.950 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.523804+0000" 2024-09-22T03:10:23.950 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.950 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.950 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.950 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.950 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.950 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.950 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:10:23.950 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.950 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.951 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.951 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.951 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.951 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.951 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.951 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:10:23.951 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.524941+0000" 2024-09-22T03:10:23.951 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.951 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.951 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.951 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.951 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.951 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.951 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:10:23.951 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.952 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.952 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.952 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.952 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.952 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.952 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.952 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:10:23.952 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.952 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.952 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.952 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.952 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.952 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.952 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.952 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:10:23.953 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.953 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.953 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.953 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.953 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.953 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.953 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.953 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:10:23.953 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.525313+0000" 2024-09-22T03:10:23.953 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.953 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.953 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.953 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.953 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.954 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.954 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:10:23.954 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.954 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.954 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.954 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.954 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.954 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.954 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.954 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:10:23.954 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.527127+0000" 2024-09-22T03:10:23.954 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.954 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.954 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.955 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.955 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.955 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.955 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:10:23.955 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.955 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.955 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.955 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.955 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.955 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.955 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.955 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:10:23.955 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.571526+0000" 2024-09-22T03:10:23.955 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.955 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.956 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.956 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.956 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.956 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.956 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:10:23.956 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.956 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:10:23.956 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:10:23.956 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.956 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-22T03:10:23.956 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "client_request", 2024-09-22T03:10:23.956 INFO:teuthology.orchestra.run.smithi153.stdout: "id": 582, 2024-09-22T03:10:23.956 INFO:teuthology.orchestra.run.smithi153.stdout: "detail": { 2024-09-22T03:10:23.957 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.957 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:10:23.957 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.527545+0000" 2024-09-22T03:10:23.957 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.957 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.957 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:10:23.957 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.527550+0000" 2024-09-22T03:10:23.957 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.957 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.957 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.957 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.957 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.957 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.957 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:10:23.957 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.527869+0000" 2024-09-22T03:10:23.958 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.958 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.958 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.958 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.958 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.958 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.958 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:10:23.958 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.958 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.958 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.958 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.958 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.958 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.958 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.958 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:10:23.959 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.530235+0000" 2024-09-22T03:10:23.959 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.959 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.959 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.959 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.959 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.959 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.959 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:10:23.959 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.530641+0000" 2024-09-22T03:10:23.959 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.959 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.959 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.959 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.959 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.960 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.960 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:10:23.960 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.960 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.960 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.960 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.960 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.960 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.960 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.960 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:10:23.960 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.572732+0000" 2024-09-22T03:10:23.960 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.960 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.960 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:10:23.960 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.531603+0000" 2024-09-22T03:10:23.961 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.961 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.961 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.961 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.961 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.961 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.961 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:10:23.961 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.961 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.961 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.961 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.961 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.961 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.961 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.961 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:10:23.962 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.531936+0000" 2024-09-22T03:10:23.962 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.962 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.962 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.962 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.962 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.962 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.962 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:10:23.962 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.962 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.962 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.962 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.963 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.963 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.963 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.963 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:10:23.963 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.532803+0000" 2024-09-22T03:10:23.963 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.963 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.963 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.963 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.963 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.963 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.963 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:10:23.963 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.963 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.963 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.964 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.964 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.964 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.964 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.964 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:10:23.964 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.964 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.964 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.964 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.964 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.964 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.964 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.964 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:10:23.964 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.964 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.965 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.965 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.965 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.965 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.965 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.965 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:10:23.965 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.533201+0000" 2024-09-22T03:10:23.965 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.965 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.965 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.965 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.965 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.965 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.966 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:10:23.966 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.966 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.966 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.966 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.966 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.966 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.966 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.966 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:10:23.966 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.534764+0000" 2024-09-22T03:10:23.966 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.966 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.966 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.966 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.967 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.967 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.967 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:10:23.967 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.967 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.967 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.967 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.967 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.967 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.967 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.967 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:10:23.967 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.572418+0000" 2024-09-22T03:10:23.967 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.967 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.967 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.968 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.968 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.968 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.968 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:10:23.968 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.968 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:10:23.968 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:10:23.968 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.968 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-22T03:10:23.968 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "client_request", 2024-09-22T03:10:23.968 INFO:teuthology.orchestra.run.smithi153.stdout: "id": 583, 2024-09-22T03:10:23.968 INFO:teuthology.orchestra.run.smithi153.stdout: "detail": { 2024-09-22T03:10:23.968 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.968 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:10:23.969 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.535274+0000" 2024-09-22T03:10:23.969 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.969 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.969 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:10:23.969 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.535280+0000" 2024-09-22T03:10:23.969 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.969 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.969 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.969 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.969 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.969 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.969 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:10:23.969 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.536695+0000" 2024-09-22T03:10:23.969 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.969 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.970 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.970 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.970 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.970 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.970 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:10:23.970 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.970 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.970 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.970 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.970 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.970 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.970 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.970 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:10:23.970 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.537096+0000" 2024-09-22T03:10:23.971 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.971 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.971 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.971 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.971 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.971 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.971 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:10:23.971 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.537444+0000" 2024-09-22T03:10:23.971 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.971 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.971 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.971 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.971 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.971 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.971 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:10:23.972 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.972 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.972 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.972 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.972 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.972 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.972 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.972 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:10:23.972 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.574017+0000" 2024-09-22T03:10:23.972 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.972 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.972 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:10:23.972 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.537952+0000" 2024-09-22T03:10:23.972 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.972 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.973 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.973 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.973 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.973 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.973 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:10:23.973 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.973 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.973 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.973 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.973 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.973 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.973 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.973 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:10:23.973 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.538559+0000" 2024-09-22T03:10:23.974 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.974 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.974 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.974 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.974 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.974 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.974 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:10:23.974 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.974 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.974 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.974 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.974 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.974 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.974 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.974 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:10:23.975 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.539528+0000" 2024-09-22T03:10:23.975 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.975 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.975 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.975 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.975 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.975 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.975 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:10:23.975 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.975 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.975 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.975 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.975 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.975 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.976 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.976 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:10:23.976 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.976 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.976 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.976 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.976 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.976 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.976 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.976 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:10:23.976 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.976 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.976 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.976 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.977 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.977 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.977 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.977 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:10:23.977 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.541094+0000" 2024-09-22T03:10:23.977 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.977 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.977 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.977 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.977 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.977 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.977 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:10:23.977 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.977 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.977 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.978 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.978 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.978 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.978 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.978 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:10:23.978 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.541846+0000" 2024-09-22T03:10:23.978 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.978 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.978 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.978 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.978 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.978 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.978 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:10:23.978 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.979 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.979 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.979 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.979 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.979 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.979 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.979 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:10:23.979 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.573520+0000" 2024-09-22T03:10:23.979 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.979 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.979 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.979 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.979 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.979 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.980 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:10:23.980 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.980 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:10:23.980 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:10:23.980 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.980 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-22T03:10:23.980 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "client_request", 2024-09-22T03:10:23.980 INFO:teuthology.orchestra.run.smithi153.stdout: "id": 586, 2024-09-22T03:10:23.980 INFO:teuthology.orchestra.run.smithi153.stdout: "detail": { 2024-09-22T03:10:23.980 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.980 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:10:23.980 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.542865+0000" 2024-09-22T03:10:23.980 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.980 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.981 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:10:23.981 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.542871+0000" 2024-09-22T03:10:23.981 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.981 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.981 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.981 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.981 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.981 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.981 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:10:23.981 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.544803+0000" 2024-09-22T03:10:23.981 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.981 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.981 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.981 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.982 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.982 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.982 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:10:23.982 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.982 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.982 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.982 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.982 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.982 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.982 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.982 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:10:23.982 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.545055+0000" 2024-09-22T03:10:23.982 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.983 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.983 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.983 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.983 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.983 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.983 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:10:23.983 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.545810+0000" 2024-09-22T03:10:23.983 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.983 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.983 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.983 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.983 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.983 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.983 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:10:23.984 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.984 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.984 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.984 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.984 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.984 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.984 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.984 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:10:23.984 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.578904+0000" 2024-09-22T03:10:23.984 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.984 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.984 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:10:23.984 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.547071+0000" 2024-09-22T03:10:23.984 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.985 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.985 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.985 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.985 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.985 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.985 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:10:23.985 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.985 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.985 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.985 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.985 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.985 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.985 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.985 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:10:23.986 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.548633+0000" 2024-09-22T03:10:23.986 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.986 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.986 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.986 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.986 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.986 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.986 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:10:23.986 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.986 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.986 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.986 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.986 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.986 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.987 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.987 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:10:23.987 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.548927+0000" 2024-09-22T03:10:23.987 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.987 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.987 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.987 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.987 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.987 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.987 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:10:23.987 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.987 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.987 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.987 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.988 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.988 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.988 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.988 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:10:23.988 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.988 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.988 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.988 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.988 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.988 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.988 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.988 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:10:23.988 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.988 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.989 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.989 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.989 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.989 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.989 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.989 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:10:23.989 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.549297+0000" 2024-09-22T03:10:23.989 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.989 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.989 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.989 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.989 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.989 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.989 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:10:23.990 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.990 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.990 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.990 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.990 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.990 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.990 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.990 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:10:23.990 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.550161+0000" 2024-09-22T03:10:23.990 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.990 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.990 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.990 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.990 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.991 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.991 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:10:23.991 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.991 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.991 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.991 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.991 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.991 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.991 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.991 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:10:23.991 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.577986+0000" 2024-09-22T03:10:23.991 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.991 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.991 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.992 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.992 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.992 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.992 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:10:23.992 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.992 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:10:23.992 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:10:23.992 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.992 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-22T03:10:23.992 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "client_request", 2024-09-22T03:10:23.992 INFO:teuthology.orchestra.run.smithi153.stdout: "id": 587, 2024-09-22T03:10:23.992 INFO:teuthology.orchestra.run.smithi153.stdout: "detail": { 2024-09-22T03:10:23.992 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.992 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:10:23.992 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.549429+0000" 2024-09-22T03:10:23.993 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.993 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.993 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:10:23.993 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.549439+0000" 2024-09-22T03:10:23.993 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.993 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.993 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.993 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.993 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.993 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.993 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:10:23.993 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.550235+0000" 2024-09-22T03:10:23.993 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.993 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.993 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.994 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.994 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.994 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.994 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:10:23.994 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.994 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.994 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.994 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.994 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.994 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.994 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.994 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:10:23.994 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.551576+0000" 2024-09-22T03:10:23.995 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.995 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.995 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.995 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.995 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.995 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.995 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:10:23.995 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.553383+0000" 2024-09-22T03:10:23.995 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.995 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.995 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.995 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.995 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.995 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.995 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:10:23.996 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.996 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.996 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.996 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.996 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.996 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.996 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.996 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:10:23.996 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.583559+0000" 2024-09-22T03:10:23.996 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.996 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.996 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:10:23.996 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.553933+0000" 2024-09-22T03:10:23.996 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.996 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.997 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.997 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.997 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.997 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.997 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:10:23.997 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.997 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.997 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.997 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.997 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.997 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.997 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.997 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:10:23.997 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.554152+0000" 2024-09-22T03:10:23.997 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.998 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.998 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.998 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.998 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.998 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.998 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:10:23.998 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.998 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.998 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.998 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.998 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.998 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.998 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.998 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:10:23.999 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.554434+0000" 2024-09-22T03:10:23.999 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.999 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.999 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.999 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.999 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.999 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:23.999 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:10:23.999 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.999 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:23.999 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:23.999 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:23.999 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:23.999 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.000 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.000 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:10:24.000 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.000 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.000 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.000 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.000 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.000 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.000 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.000 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:10:24.000 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.000 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.000 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.000 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.000 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.001 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.001 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.001 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:10:24.001 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.555799+0000" 2024-09-22T03:10:24.001 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.001 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.001 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.001 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.001 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.001 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.001 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:10:24.001 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.001 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.002 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.002 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.002 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.002 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.002 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.002 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:10:24.002 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.556737+0000" 2024-09-22T03:10:24.002 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.002 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.002 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.002 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.002 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.002 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.002 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:10:24.003 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.003 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.003 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.003 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.003 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.003 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.003 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.003 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:10:24.003 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.582679+0000" 2024-09-22T03:10:24.003 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.003 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.003 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.003 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.004 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.004 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.004 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:10:24.004 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.004 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:10:24.004 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:10:24.004 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.004 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-22T03:10:24.004 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "client_request", 2024-09-22T03:10:24.004 INFO:teuthology.orchestra.run.smithi153.stdout: "id": 625, 2024-09-22T03:10:24.004 INFO:teuthology.orchestra.run.smithi153.stdout: "detail": { 2024-09-22T03:10:24.004 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.004 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:10:24.004 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.681810+0000" 2024-09-22T03:10:24.004 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.005 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.005 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:10:24.005 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.681826+0000" 2024-09-22T03:10:24.005 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.005 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.005 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.005 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.005 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.005 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.005 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:10:24.005 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.681991+0000" 2024-09-22T03:10:24.005 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.005 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.005 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.005 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.006 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.006 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.006 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:10:24.006 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.006 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.006 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.006 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.006 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.006 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.006 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.006 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:10:24.006 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.682117+0000" 2024-09-22T03:10:24.006 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.006 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.006 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.007 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.007 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.007 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.007 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:10:24.007 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.685405+0000" 2024-09-22T03:10:24.007 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.007 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.007 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.007 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.007 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.007 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.007 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:10:24.007 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.007 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.008 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.008 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.008 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.008 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.008 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.008 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:10:24.008 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.008 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.008 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.008 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:10:24.008 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.686133+0000" 2024-09-22T03:10:24.008 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.008 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.008 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.009 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.009 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.009 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.009 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:10:24.009 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.009 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.009 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.009 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.009 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.009 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.009 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.009 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:10:24.009 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.686387+0000" 2024-09-22T03:10:24.009 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.009 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.010 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.010 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.010 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.010 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.010 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:10:24.010 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.010 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.010 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.010 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.010 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.010 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.010 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.011 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:10:24.011 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.686928+0000" 2024-09-22T03:10:24.011 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.011 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.011 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.011 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.011 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.011 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.011 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:10:24.011 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.011 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.011 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.011 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.011 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.012 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.012 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.012 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:10:24.012 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.012 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.012 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.012 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.012 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.012 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.012 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.012 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:10:24.012 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.012 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.012 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.012 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.013 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.013 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.013 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.013 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:10:24.013 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.687195+0000" 2024-09-22T03:10:24.013 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.013 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.013 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.013 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.013 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.013 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.013 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:10:24.013 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.013 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.013 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.014 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.014 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.014 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.014 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.014 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:10:24.014 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.728675+0000" 2024-09-22T03:10:24.014 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.014 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.014 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.014 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.014 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.014 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.014 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:10:24.014 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.015 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.015 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.015 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.015 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.015 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.015 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.015 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:10:24.015 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.015 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.015 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.015 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.015 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.015 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.015 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.015 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:10:24.016 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.016 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:10:24.016 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:10:24.016 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.016 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-22T03:10:24.016 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "client_request", 2024-09-22T03:10:24.016 INFO:teuthology.orchestra.run.smithi153.stdout: "id": 627, 2024-09-22T03:10:24.016 INFO:teuthology.orchestra.run.smithi153.stdout: "detail": { 2024-09-22T03:10:24.016 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.016 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:10:24.016 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.689029+0000" 2024-09-22T03:10:24.016 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.016 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.016 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:10:24.016 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.689036+0000" 2024-09-22T03:10:24.017 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.017 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.017 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.017 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.017 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.017 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.017 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:10:24.017 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.689133+0000" 2024-09-22T03:10:24.017 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.017 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.017 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.017 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.017 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.017 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.018 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:10:24.018 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.018 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.018 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.018 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.018 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.018 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.018 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.018 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:10:24.018 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.689256+0000" 2024-09-22T03:10:24.018 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.018 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.018 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.018 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.018 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.019 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.019 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:10:24.019 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.692076+0000" 2024-09-22T03:10:24.019 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.019 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.019 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.019 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.019 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.019 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.019 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:10:24.019 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.019 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.019 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.019 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.019 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.020 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.020 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.020 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:10:24.020 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.020 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.020 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.020 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:10:24.020 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.692491+0000" 2024-09-22T03:10:24.020 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.020 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.020 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.020 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.020 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.020 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.020 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:10:24.021 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.021 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.021 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.021 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.021 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.021 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.021 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.021 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:10:24.021 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.692686+0000" 2024-09-22T03:10:24.021 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.021 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.021 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.021 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.022 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.022 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.022 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:10:24.022 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.022 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.022 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.022 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.022 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.022 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.022 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.022 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:10:24.022 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.693187+0000" 2024-09-22T03:10:24.022 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.022 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.022 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.023 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.023 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.023 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.023 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:10:24.023 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.023 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.023 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.023 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.023 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.023 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.023 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.023 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:10:24.023 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.023 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.024 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.024 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.024 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.024 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.024 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.024 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:10:24.024 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.024 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.024 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.024 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.024 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.024 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.024 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.024 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:10:24.024 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.693639+0000" 2024-09-22T03:10:24.025 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.025 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.025 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.025 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.025 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.025 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.025 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:10:24.025 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.025 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.025 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.025 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.025 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.728664+0000" 2024-09-22T03:10:24.025 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.025 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.025 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:10:24.026 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.729056+0000", 2024-09-22T03:10:24.026 INFO:teuthology.orchestra.run.smithi153.stdout: "blocker": { 2024-09-22T03:10:24.026 INFO:teuthology.orchestra.run.smithi153.stdout: "op_type": "CommonPGPipeline::process", 2024-09-22T03:10:24.026 INFO:teuthology.orchestra.run.smithi153.stdout: "detail": { 2024-09-22T03:10:24.026 INFO:teuthology.orchestra.run.smithi153.stdout: "waiting": 1, 2024-09-22T03:10:24.026 INFO:teuthology.orchestra.run.smithi153.stdout: "held_by_operation_id": 625 2024-09-22T03:10:24.026 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:10:24.026 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:10:24.026 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.026 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.026 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.026 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.026 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.026 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.027 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:10:24.027 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.027 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.027 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.027 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.027 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.027 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.027 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.027 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:10:24.027 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.027 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.027 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.027 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.027 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.027 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.028 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.028 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:10:24.028 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.028 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:10:24.028 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:10:24.028 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.028 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-22T03:10:24.028 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "client_request", 2024-09-22T03:10:24.028 INFO:teuthology.orchestra.run.smithi153.stdout: "id": 628, 2024-09-22T03:10:24.028 INFO:teuthology.orchestra.run.smithi153.stdout: "detail": { 2024-09-22T03:10:24.028 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.028 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:10:24.028 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.690946+0000" 2024-09-22T03:10:24.028 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.029 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.029 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:10:24.029 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.690958+0000" 2024-09-22T03:10:24.029 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.029 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.029 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.029 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.029 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.029 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.029 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:10:24.029 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.691115+0000" 2024-09-22T03:10:24.029 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.029 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.029 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.029 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.030 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.030 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.030 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:10:24.030 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.030 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.030 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.030 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.030 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.030 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.030 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.030 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:10:24.030 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.691216+0000" 2024-09-22T03:10:24.030 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.030 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.030 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.031 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.031 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.031 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.031 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:10:24.031 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.692157+0000" 2024-09-22T03:10:24.031 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.031 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.031 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.031 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.031 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.031 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.031 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:10:24.031 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.031 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.031 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.032 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.032 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.032 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.032 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.032 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:10:24.032 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.032 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.032 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.032 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:10:24.032 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.693007+0000" 2024-09-22T03:10:24.032 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.032 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.033 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.033 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.033 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.033 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.033 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:10:24.033 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.033 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.033 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.033 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.033 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.033 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.033 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.033 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:10:24.033 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.693331+0000" 2024-09-22T03:10:24.033 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.034 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.034 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.034 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.034 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.034 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.034 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:10:24.034 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.034 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.034 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.034 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.034 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.034 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.034 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.034 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:10:24.035 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.694060+0000" 2024-09-22T03:10:24.035 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.035 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.035 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.035 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.035 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.035 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.035 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:10:24.035 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.035 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.035 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.035 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.035 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.035 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.035 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.036 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:10:24.036 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.036 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.036 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.036 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.036 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.036 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.036 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.036 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:10:24.036 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.036 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.036 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.036 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.036 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.036 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.037 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.037 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:10:24.037 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.694393+0000" 2024-09-22T03:10:24.037 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.037 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.037 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.037 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.037 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.037 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.037 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:10:24.037 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.037 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.037 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.037 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.037 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.728265+0000" 2024-09-22T03:10:24.038 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.038 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.038 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:10:24.038 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.038 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.038 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.038 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.038 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.038 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.038 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.038 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:10:24.038 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.038 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.038 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.039 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.039 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.039 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.039 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.039 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:10:24.039 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.039 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.039 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.039 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.039 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.039 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.039 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.039 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:10:24.039 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.039 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:10:24.040 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:10:24.040 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.040 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-22T03:10:24.040 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "client_request", 2024-09-22T03:10:24.040 INFO:teuthology.orchestra.run.smithi153.stdout: "id": 631, 2024-09-22T03:10:24.040 INFO:teuthology.orchestra.run.smithi153.stdout: "detail": { 2024-09-22T03:10:24.040 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.040 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:10:24.040 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.700424+0000" 2024-09-22T03:10:24.040 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.040 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.040 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:10:24.040 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.700439+0000" 2024-09-22T03:10:24.040 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.040 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.041 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.041 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.041 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.041 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.041 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:10:24.041 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.700621+0000" 2024-09-22T03:10:24.041 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.041 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.041 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.041 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.041 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.041 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.041 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:10:24.041 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.041 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.042 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.042 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.042 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.042 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.042 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.042 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:10:24.042 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.700711+0000" 2024-09-22T03:10:24.042 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.042 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.042 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.042 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.042 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.042 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.042 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:10:24.043 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.701350+0000" 2024-09-22T03:10:24.043 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.043 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.043 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.043 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.043 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.043 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.043 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:10:24.043 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.043 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.043 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.043 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.043 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.043 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.044 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.044 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:10:24.044 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.044 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.044 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.044 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:10:24.044 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.701779+0000" 2024-09-22T03:10:24.044 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.044 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.044 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.044 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.044 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.044 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.044 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:10:24.044 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.045 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.045 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.045 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.045 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.045 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.045 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.045 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:10:24.045 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.701967+0000" 2024-09-22T03:10:24.045 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.045 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.045 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.045 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.045 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.045 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.046 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:10:24.046 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.046 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.046 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.046 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.046 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.046 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.046 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.046 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:10:24.046 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.702186+0000" 2024-09-22T03:10:24.046 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.046 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.046 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.046 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.047 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.047 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.047 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:10:24.047 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.047 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.047 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.047 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.047 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.047 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.047 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.047 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:10:24.047 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.047 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.047 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.048 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.048 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.048 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.048 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.048 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:10:24.048 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.048 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.048 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.048 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.048 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.048 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.048 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.048 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:10:24.048 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.702952+0000" 2024-09-22T03:10:24.048 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.049 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.049 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.049 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.049 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.049 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.049 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:10:24.049 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.049 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.049 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.049 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.049 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.728508+0000" 2024-09-22T03:10:24.049 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.049 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.049 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:10:24.049 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.050 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.050 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.050 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.050 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.050 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.050 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.050 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:10:24.050 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.050 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.050 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.050 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.050 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.050 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.050 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.051 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:10:24.051 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.051 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.051 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.051 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.051 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.051 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.051 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.051 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:10:24.051 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.051 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:10:24.051 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:10:24.051 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.051 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-22T03:10:24.051 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "client_request", 2024-09-22T03:10:24.052 INFO:teuthology.orchestra.run.smithi153.stdout: "id": 633, 2024-09-22T03:10:24.052 INFO:teuthology.orchestra.run.smithi153.stdout: "detail": { 2024-09-22T03:10:24.052 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.052 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:10:24.052 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.704768+0000" 2024-09-22T03:10:24.052 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.052 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.052 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:10:24.052 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.704780+0000" 2024-09-22T03:10:24.052 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.052 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.052 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.052 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.052 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.052 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.053 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:10:24.053 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.704931+0000" 2024-09-22T03:10:24.053 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.053 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.053 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.053 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.053 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.053 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.053 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:10:24.053 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.053 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.053 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.053 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.053 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.053 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.054 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.054 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:10:24.054 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.705015+0000" 2024-09-22T03:10:24.054 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.054 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.054 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.054 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.054 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.054 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.054 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:10:24.055 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.705507+0000" 2024-09-22T03:10:24.055 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.055 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.055 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.055 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.055 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.055 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.055 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:10:24.055 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.055 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.055 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.055 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.055 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.055 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.055 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.056 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:10:24.056 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.056 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.056 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.056 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:10:24.056 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.705899+0000" 2024-09-22T03:10:24.056 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.056 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.056 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.056 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.056 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.056 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.056 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:10:24.056 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.056 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.057 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.057 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.057 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.057 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.057 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.057 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:10:24.057 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.706153+0000" 2024-09-22T03:10:24.057 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.057 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.057 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.057 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.057 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.057 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.057 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:10:24.058 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.058 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.058 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.058 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.058 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.058 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.058 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.058 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:10:24.058 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.706323+0000" 2024-09-22T03:10:24.058 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.058 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.058 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.058 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.058 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.058 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.059 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:10:24.059 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.059 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.059 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.059 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.059 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.059 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.059 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.059 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:10:24.059 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.059 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.059 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.059 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.059 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.059 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.060 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.060 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:10:24.060 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.060 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.060 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.060 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.060 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.060 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.060 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.060 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:10:24.060 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.706580+0000" 2024-09-22T03:10:24.060 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.060 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.060 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.060 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.061 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.061 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.061 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:10:24.061 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.061 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.061 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.061 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.061 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.728562+0000" 2024-09-22T03:10:24.061 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.061 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.061 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:10:24.061 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.061 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.061 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.062 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.062 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.062 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.062 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.062 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:10:24.062 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.062 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.062 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.062 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.062 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.062 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.062 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.062 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:10:24.063 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.063 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.063 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.063 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.063 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.063 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.063 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.063 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:10:24.063 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.063 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:10:24.063 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:10:24.063 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.063 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-22T03:10:24.063 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "client_request", 2024-09-22T03:10:24.063 INFO:teuthology.orchestra.run.smithi153.stdout: "id": 635, 2024-09-22T03:10:24.064 INFO:teuthology.orchestra.run.smithi153.stdout: "detail": { 2024-09-22T03:10:24.064 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.064 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:10:24.064 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.711613+0000" 2024-09-22T03:10:24.064 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.064 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.064 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:10:24.064 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.711623+0000" 2024-09-22T03:10:24.064 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.064 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.064 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.064 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.064 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.064 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.065 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:10:24.065 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.711739+0000" 2024-09-22T03:10:24.065 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.065 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.065 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.065 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.065 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.065 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.065 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:10:24.065 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.065 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.065 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.065 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.065 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.066 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.066 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.066 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:10:24.066 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.711845+0000" 2024-09-22T03:10:24.066 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.066 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.066 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.066 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.066 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.066 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.066 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:10:24.066 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.714317+0000" 2024-09-22T03:10:24.066 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.066 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.067 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.067 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.067 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.067 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.067 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:10:24.067 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.067 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.067 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.067 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.067 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.067 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.067 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.067 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:10:24.067 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.067 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.068 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.068 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:10:24.068 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.714765+0000" 2024-09-22T03:10:24.068 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.068 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.068 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.068 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.068 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.068 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.068 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:10:24.068 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.068 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.068 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.068 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.068 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.069 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.069 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.069 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:10:24.069 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.714971+0000" 2024-09-22T03:10:24.069 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.069 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.069 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.069 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.069 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.069 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.069 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:10:24.069 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.069 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.069 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.069 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.070 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.070 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.070 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.070 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:10:24.070 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.715155+0000" 2024-09-22T03:10:24.070 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.070 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.070 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.070 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.070 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.070 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.070 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:10:24.070 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.070 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.071 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.071 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.071 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.071 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.071 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.071 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:10:24.071 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.071 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.071 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.071 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.071 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.071 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.071 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.071 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:10:24.071 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.072 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.072 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.072 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.072 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.072 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.072 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.072 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:10:24.072 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.715439+0000" 2024-09-22T03:10:24.072 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.072 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.072 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.072 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.072 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.072 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.072 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:10:24.073 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.073 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.073 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.073 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.073 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.728350+0000" 2024-09-22T03:10:24.073 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.073 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.073 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:10:24.073 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.073 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.073 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.073 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.073 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.073 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.073 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.074 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:10:24.074 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.074 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.074 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.074 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.074 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.074 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.074 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.074 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:10:24.074 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.074 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.074 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.074 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.074 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.075 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.075 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.075 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:10:24.075 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.075 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:10:24.075 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:10:24.075 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.075 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-22T03:10:24.075 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "client_request", 2024-09-22T03:10:24.075 INFO:teuthology.orchestra.run.smithi153.stdout: "id": 636, 2024-09-22T03:10:24.075 INFO:teuthology.orchestra.run.smithi153.stdout: "detail": { 2024-09-22T03:10:24.075 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.075 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:10:24.075 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.716796+0000" 2024-09-22T03:10:24.075 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.076 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.076 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:10:24.076 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.716807+0000" 2024-09-22T03:10:24.076 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.076 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.076 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.076 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.076 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.076 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.076 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:10:24.076 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.717204+0000" 2024-09-22T03:10:24.076 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.076 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.077 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.077 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.077 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.077 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.077 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:10:24.077 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.077 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.077 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.077 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.077 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.077 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.077 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.077 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:10:24.078 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.717342+0000" 2024-09-22T03:10:24.078 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.078 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.078 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.078 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.078 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.078 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.078 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:10:24.078 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.717643+0000" 2024-09-22T03:10:24.078 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.078 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.078 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.078 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.078 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.079 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.079 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:10:24.079 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.079 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.079 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.079 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.079 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.079 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.079 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.079 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:10:24.079 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.079 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.079 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.079 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:10:24.080 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.718065+0000" 2024-09-22T03:10:24.080 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.080 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.080 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.080 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.080 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.080 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.080 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:10:24.080 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.080 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.080 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.080 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.080 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.080 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.081 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.081 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:10:24.081 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.718249+0000" 2024-09-22T03:10:24.081 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.081 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.081 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.081 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.081 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.081 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.081 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:10:24.081 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.081 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.081 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.081 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.082 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.082 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.082 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.082 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:10:24.082 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.718426+0000" 2024-09-22T03:10:24.082 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.082 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.082 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.082 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.082 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.082 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.082 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:10:24.082 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.082 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.083 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.083 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.083 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.083 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.083 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.083 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:10:24.083 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.083 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.083 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.083 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.083 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.083 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.083 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.083 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:10:24.084 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.084 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.084 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.084 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.084 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.084 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.084 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.084 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:10:24.084 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.718689+0000" 2024-09-22T03:10:24.084 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.084 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.084 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.084 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.084 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.085 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.085 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:10:24.085 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.085 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.085 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.085 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.085 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.728453+0000" 2024-09-22T03:10:24.085 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.085 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.085 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:10:24.085 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.085 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.085 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.085 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.086 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.086 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.086 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.086 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:10:24.086 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.086 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.086 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.086 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.086 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.086 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.086 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.086 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:10:24.086 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.086 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.087 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.087 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.087 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.087 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.087 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.087 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:10:24.087 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.087 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:10:24.087 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:10:24.087 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.087 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-22T03:10:24.087 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "client_request", 2024-09-22T03:10:24.087 INFO:teuthology.orchestra.run.smithi153.stdout: "id": 637, 2024-09-22T03:10:24.087 INFO:teuthology.orchestra.run.smithi153.stdout: "detail": { 2024-09-22T03:10:24.088 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.088 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:10:24.088 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.721648+0000" 2024-09-22T03:10:24.088 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.088 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.088 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:10:24.088 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.721655+0000" 2024-09-22T03:10:24.088 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.088 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.088 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.088 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.088 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.088 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.088 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:10:24.088 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.721738+0000" 2024-09-22T03:10:24.089 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.089 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.089 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.089 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.089 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.089 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.089 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:10:24.089 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.089 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.089 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.089 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.089 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.089 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.089 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.089 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:10:24.090 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.721787+0000" 2024-09-22T03:10:24.090 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.090 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.090 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.090 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.090 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.090 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.090 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:10:24.090 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.721936+0000" 2024-09-22T03:10:24.090 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.090 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.090 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.090 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.090 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.091 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.091 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:10:24.091 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.091 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.091 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.091 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.091 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.091 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.091 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.091 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:10:24.091 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.091 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.091 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.091 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:10:24.091 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.722293+0000" 2024-09-22T03:10:24.092 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.092 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.092 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.092 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.092 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.092 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.092 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:10:24.092 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.092 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.092 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.092 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.092 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.092 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.092 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.093 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:10:24.093 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.722474+0000" 2024-09-22T03:10:24.093 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.093 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.093 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.093 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.093 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.093 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.093 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:10:24.093 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.093 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.093 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.093 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.093 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.093 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.094 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.094 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:10:24.094 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.722634+0000" 2024-09-22T03:10:24.094 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.094 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.094 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.094 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.094 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.094 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.094 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:10:24.094 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.094 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.094 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.094 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.095 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.095 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.095 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.095 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:10:24.095 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.095 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.095 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.095 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.095 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.095 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.095 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.095 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:10:24.095 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.095 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.095 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.096 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.096 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.096 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.096 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.096 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:10:24.096 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.722880+0000" 2024-09-22T03:10:24.096 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.096 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.096 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.096 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.096 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.096 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.096 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:10:24.096 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.096 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.097 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.097 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.097 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.724731+0000" 2024-09-22T03:10:24.097 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.097 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.097 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:10:24.097 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.097 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.097 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.097 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.097 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.097 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.097 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.097 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:10:24.098 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.098 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.098 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.098 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.098 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.098 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.098 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.098 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:10:24.098 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.098 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.098 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.098 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.098 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.099 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.099 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.099 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:10:24.099 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.099 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:10:24.099 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:10:24.099 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.099 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-22T03:10:24.099 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "client_request", 2024-09-22T03:10:24.099 INFO:teuthology.orchestra.run.smithi153.stdout: "id": 639, 2024-09-22T03:10:24.099 INFO:teuthology.orchestra.run.smithi153.stdout: "detail": { 2024-09-22T03:10:24.099 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.099 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:10:24.099 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.722859+0000" 2024-09-22T03:10:24.099 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.100 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.100 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:10:24.100 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.722863+0000" 2024-09-22T03:10:24.100 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.100 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.100 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.100 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.100 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.100 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.100 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:10:24.100 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.722929+0000" 2024-09-22T03:10:24.100 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.100 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.100 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.101 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.101 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.101 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.101 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:10:24.101 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.101 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.101 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.101 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.101 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.101 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.101 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.101 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:10:24.101 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.722982+0000" 2024-09-22T03:10:24.101 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.101 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.102 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.102 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.102 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.102 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.102 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:10:24.102 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.723485+0000" 2024-09-22T03:10:24.102 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.102 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.102 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.102 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.102 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.102 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.102 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:10:24.102 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.103 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.103 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.103 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.103 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.103 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.103 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.103 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:10:24.103 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.103 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.103 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.103 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:10:24.103 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.724251+0000" 2024-09-22T03:10:24.103 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.103 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.103 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.104 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.104 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.104 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.104 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:10:24.104 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.104 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.104 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.104 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.104 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.104 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.104 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.104 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:10:24.104 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.725268+0000" 2024-09-22T03:10:24.104 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.104 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.105 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.105 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.105 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.105 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.105 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:10:24.105 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.105 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.105 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.105 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.105 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.105 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.105 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.105 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:10:24.105 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.725936+0000" 2024-09-22T03:10:24.106 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.106 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.106 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.106 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.106 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.106 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.106 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:10:24.106 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.106 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.106 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.106 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.106 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.106 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.106 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.106 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:10:24.107 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.107 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.107 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.107 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.107 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.107 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.107 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.107 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:10:24.107 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.107 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.107 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.107 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.107 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.107 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.108 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.108 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:10:24.108 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.726164+0000" 2024-09-22T03:10:24.108 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.108 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.108 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.108 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.108 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.108 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.108 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:10:24.108 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.108 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.108 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.108 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.108 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.726443+0000" 2024-09-22T03:10:24.109 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.109 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.109 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:10:24.109 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.109 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.109 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.109 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.109 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.109 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.109 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.109 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:10:24.109 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.109 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.109 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.110 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.110 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.110 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.110 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.110 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:10:24.110 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.110 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.110 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.110 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.110 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.110 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.110 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.110 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:10:24.110 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.110 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:10:24.111 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:10:24.111 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.111 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-22T03:10:24.111 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "client_request", 2024-09-22T03:10:24.111 INFO:teuthology.orchestra.run.smithi153.stdout: "id": 638, 2024-09-22T03:10:24.111 INFO:teuthology.orchestra.run.smithi153.stdout: "detail": { 2024-09-22T03:10:24.111 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.111 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:10:24.111 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.722428+0000" 2024-09-22T03:10:24.111 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.111 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.111 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:10:24.111 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.722433+0000" 2024-09-22T03:10:24.111 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.111 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.112 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.112 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.112 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.112 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.112 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:10:24.112 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.722543+0000" 2024-09-22T03:10:24.112 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.112 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.112 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.112 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.112 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.112 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.112 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:10:24.112 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.113 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.113 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.113 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.113 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.113 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.113 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.113 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:10:24.113 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.722603+0000" 2024-09-22T03:10:24.113 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.113 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.113 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.113 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.113 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.113 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.113 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:10:24.114 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.723437+0000" 2024-09-22T03:10:24.114 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.114 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.114 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.114 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.114 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.114 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.114 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:10:24.114 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.114 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.114 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.114 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.114 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.114 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.115 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.115 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:10:24.115 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.115 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.115 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.115 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:10:24.115 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.723823+0000" 2024-09-22T03:10:24.115 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.115 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.115 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.115 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.115 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.115 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.115 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:10:24.115 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.116 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.116 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.116 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.116 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.116 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.116 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.116 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:10:24.116 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.724328+0000" 2024-09-22T03:10:24.116 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.116 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.116 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.116 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.116 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.116 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.117 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:10:24.117 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.117 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.117 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.117 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.117 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.117 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.117 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.117 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:10:24.117 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.724942+0000" 2024-09-22T03:10:24.117 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.117 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.117 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.117 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.117 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.118 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.118 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:10:24.118 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.118 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.118 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.118 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.118 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.118 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.118 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.118 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:10:24.118 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.118 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.118 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.118 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.118 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.119 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.119 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.119 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:10:24.119 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.119 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.119 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.119 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.119 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.119 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.119 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.119 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:10:24.119 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.725343+0000" 2024-09-22T03:10:24.119 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.119 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.119 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.120 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.120 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.120 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.120 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:10:24.120 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.120 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.120 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.120 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.120 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.725793+0000" 2024-09-22T03:10:24.120 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.120 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.120 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:10:24.120 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.121 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.121 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.121 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.121 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.121 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.121 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.121 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:10:24.121 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.121 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.121 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.121 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.121 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.121 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.121 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.121 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:10:24.122 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.122 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.122 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.122 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.122 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.122 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.122 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.122 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:10:24.122 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.122 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:10:24.122 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:10:24.122 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.122 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-22T03:10:24.123 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "client_request", 2024-09-22T03:10:24.123 INFO:teuthology.orchestra.run.smithi153.stdout: "id": 640, 2024-09-22T03:10:24.123 INFO:teuthology.orchestra.run.smithi153.stdout: "detail": { 2024-09-22T03:10:24.123 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.123 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:10:24.123 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.723884+0000" 2024-09-22T03:10:24.123 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.123 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.123 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:10:24.123 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.723888+0000" 2024-09-22T03:10:24.123 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.123 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.123 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.123 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.123 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.124 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.124 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:10:24.124 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.724089+0000" 2024-09-22T03:10:24.124 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.124 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.124 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.124 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.124 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.124 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.124 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:10:24.124 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.124 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.124 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.124 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.125 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.125 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.125 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.125 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:10:24.125 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.724220+0000" 2024-09-22T03:10:24.125 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.125 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.125 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.125 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.125 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.125 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.125 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:10:24.125 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.726518+0000" 2024-09-22T03:10:24.125 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.125 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.126 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.126 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.126 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.126 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.126 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:10:24.126 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.126 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.126 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.126 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.126 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.126 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.126 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.126 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:10:24.126 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.127 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.127 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.127 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:10:24.127 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.726905+0000" 2024-09-22T03:10:24.127 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.127 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.127 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.127 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.127 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.127 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.127 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:10:24.127 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.127 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.127 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.128 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.128 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.128 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.128 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.128 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:10:24.128 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.727025+0000" 2024-09-22T03:10:24.128 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.128 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.128 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.128 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.128 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.128 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.128 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:10:24.128 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.128 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.129 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.129 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.129 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.129 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.129 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.129 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:10:24.129 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.727251+0000" 2024-09-22T03:10:24.129 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.129 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.129 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.129 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.129 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.129 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.129 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:10:24.129 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.130 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.130 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.130 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.130 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.130 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.130 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.130 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:10:24.130 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.130 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.130 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.130 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.130 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.130 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.130 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.130 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:10:24.131 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.131 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.131 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.131 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.131 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.131 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.131 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.131 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:10:24.131 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.727432+0000" 2024-09-22T03:10:24.131 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.131 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.131 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.131 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.131 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.132 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.132 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:10:24.132 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.132 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.132 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.132 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.132 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.727724+0000" 2024-09-22T03:10:24.132 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.132 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.132 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:10:24.132 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.132 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.132 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.132 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.133 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.133 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.133 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.133 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:10:24.133 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.133 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.133 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.133 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.133 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.133 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.133 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.133 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:10:24.133 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.133 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.134 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.134 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.134 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.134 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.134 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.134 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:10:24.134 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.134 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:10:24.134 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:10:24.134 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.134 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-22T03:10:24.134 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "client_request", 2024-09-22T03:10:24.134 INFO:teuthology.orchestra.run.smithi153.stdout: "id": 641, 2024-09-22T03:10:24.134 INFO:teuthology.orchestra.run.smithi153.stdout: "detail": { 2024-09-22T03:10:24.135 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.135 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:10:24.135 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.728505+0000" 2024-09-22T03:10:24.135 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.135 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.135 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:10:24.135 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.728511+0000" 2024-09-22T03:10:24.135 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.135 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.135 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.135 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.135 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.135 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.135 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:10:24.135 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.728585+0000" 2024-09-22T03:10:24.136 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.136 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.136 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.136 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.136 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.136 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.136 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:10:24.136 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.136 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.136 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.136 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.136 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.136 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.136 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.136 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:10:24.137 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.728650+0000" 2024-09-22T03:10:24.137 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.137 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.137 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.137 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.137 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.137 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.137 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:10:24.137 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.730453+0000" 2024-09-22T03:10:24.137 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.137 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.137 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.137 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.137 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.137 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.138 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:10:24.138 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.138 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.138 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.138 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.138 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.138 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.138 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.138 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:10:24.138 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.138 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.138 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.138 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:10:24.138 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.730686+0000" 2024-09-22T03:10:24.139 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.139 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.139 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.139 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.139 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.139 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.139 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:10:24.139 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.139 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.139 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.139 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.139 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.139 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.139 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.139 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:10:24.140 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.730793+0000" 2024-09-22T03:10:24.140 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.140 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.140 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.140 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.140 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.140 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.140 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:10:24.140 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.140 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.140 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.140 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.140 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.140 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.140 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.141 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:10:24.141 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.730887+0000" 2024-09-22T03:10:24.141 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.141 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.141 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.141 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.141 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.141 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.141 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:10:24.141 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.141 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.141 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.141 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.141 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.141 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.142 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.142 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:10:24.142 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.142 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.142 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.142 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.142 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.142 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.142 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.142 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:10:24.142 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.142 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.142 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.142 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.143 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.143 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.143 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.143 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:10:24.143 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.731042+0000" 2024-09-22T03:10:24.143 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.143 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.143 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.143 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.143 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.143 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.143 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:10:24.143 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.143 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.143 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.144 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.144 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.731277+0000" 2024-09-22T03:10:24.144 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.144 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.144 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:10:24.144 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.144 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.144 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.144 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.144 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.144 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.144 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.144 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:10:24.144 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.145 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.145 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.145 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.145 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.145 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.145 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.145 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:10:24.145 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.145 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.145 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.145 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.145 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.145 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.145 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.145 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:10:24.146 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.146 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:10:24.146 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:10:24.146 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.146 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-22T03:10:24.146 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "client_request", 2024-09-22T03:10:24.146 INFO:teuthology.orchestra.run.smithi153.stdout: "id": 569, 2024-09-22T03:10:24.146 INFO:teuthology.orchestra.run.smithi153.stdout: "detail": { 2024-09-22T03:10:24.146 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.146 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:10:24.146 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.470581+0000" 2024-09-22T03:10:24.146 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.146 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.146 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:10:24.147 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.470585+0000" 2024-09-22T03:10:24.149 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.149 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.149 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.149 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.150 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.150 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.150 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:10:24.150 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.470700+0000" 2024-09-22T03:10:24.150 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.150 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.150 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.150 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.150 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.150 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.150 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:10:24.150 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.150 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.150 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.151 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.151 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.151 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.151 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.151 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:10:24.151 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.470752+0000" 2024-09-22T03:10:24.151 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.151 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.151 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.151 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.151 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.151 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.151 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:10:24.151 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.471782+0000" 2024-09-22T03:10:24.152 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.152 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.152 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.152 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.152 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.152 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.152 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:10:24.152 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.152 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.152 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.152 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.152 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.152 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.152 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.152 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:10:24.153 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.567503+0000" 2024-09-22T03:10:24.153 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.153 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.153 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:10:24.153 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.472008+0000" 2024-09-22T03:10:24.153 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.153 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.153 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.153 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.153 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.153 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.153 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:10:24.153 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.153 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.153 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.154 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.154 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.154 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.154 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.154 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:10:24.154 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.472109+0000" 2024-09-22T03:10:24.154 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.154 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.154 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.154 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.154 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.154 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.154 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:10:24.154 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.155 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.155 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.155 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.155 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.155 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.155 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.155 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:10:24.155 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.472197+0000" 2024-09-22T03:10:24.155 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.155 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.155 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.155 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.155 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.155 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.156 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:10:24.156 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.156 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.156 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.156 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.156 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.156 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.156 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.156 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:10:24.156 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.156 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.156 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.156 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.156 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.156 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.157 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.157 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:10:24.157 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.157 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.157 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.157 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.157 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.157 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.157 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.157 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:10:24.157 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.472324+0000" 2024-09-22T03:10:24.157 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.157 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.157 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.158 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.158 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.158 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.158 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:10:24.158 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.158 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.158 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.158 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.158 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.474773+0000" 2024-09-22T03:10:24.158 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.158 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.158 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:10:24.158 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.487418+0000" 2024-09-22T03:10:24.158 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.158 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.159 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.159 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.159 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.159 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.159 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:10:24.159 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.159 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.159 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.159 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.159 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.159 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.159 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.159 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:10:24.159 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.567163+0000" 2024-09-22T03:10:24.160 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.160 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.160 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.160 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.160 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.160 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.160 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:10:24.160 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.160 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:10:24.160 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:10:24.160 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.160 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-22T03:10:24.160 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "client_request", 2024-09-22T03:10:24.160 INFO:teuthology.orchestra.run.smithi153.stdout: "id": 570, 2024-09-22T03:10:24.160 INFO:teuthology.orchestra.run.smithi153.stdout: "detail": { 2024-09-22T03:10:24.161 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.161 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:10:24.161 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.471497+0000" 2024-09-22T03:10:24.161 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.161 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.161 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:10:24.161 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.471503+0000" 2024-09-22T03:10:24.161 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.161 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.161 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.161 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.161 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.161 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.161 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:10:24.161 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.471611+0000" 2024-09-22T03:10:24.162 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.162 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.162 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.162 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.162 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.162 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.162 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:10:24.162 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.162 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.162 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.162 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.162 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.162 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.162 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.163 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:10:24.163 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.471881+0000" 2024-09-22T03:10:24.163 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.163 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.163 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.163 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.163 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.163 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.163 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:10:24.163 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.472535+0000" 2024-09-22T03:10:24.163 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.163 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.163 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.163 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.163 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.164 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.164 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:10:24.164 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.164 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.164 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.164 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.164 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.164 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.164 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.164 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:10:24.164 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.569410+0000" 2024-09-22T03:10:24.164 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.164 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.164 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:10:24.164 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.472727+0000" 2024-09-22T03:10:24.165 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.165 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.165 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.165 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.165 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.165 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.165 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:10:24.165 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.165 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.165 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.165 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.165 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.165 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.165 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.165 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:10:24.166 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.472818+0000" 2024-09-22T03:10:24.166 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.166 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.166 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.166 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.166 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.166 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.166 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:10:24.166 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.166 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.166 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.166 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.166 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.166 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.167 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.167 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:10:24.167 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.472922+0000" 2024-09-22T03:10:24.167 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.167 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.167 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.167 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.167 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.167 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.167 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:10:24.167 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.167 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.167 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.167 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.167 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.168 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.168 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.168 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:10:24.168 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.168 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.168 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.168 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.168 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.168 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.168 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.168 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:10:24.168 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.168 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.168 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.169 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.169 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.169 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.169 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.169 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:10:24.169 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.473047+0000" 2024-09-22T03:10:24.169 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.169 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.169 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.169 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.169 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.169 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.169 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:10:24.169 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.169 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.170 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.170 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.170 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.474888+0000" 2024-09-22T03:10:24.170 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.170 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.170 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:10:24.170 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.491124+0000" 2024-09-22T03:10:24.170 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.170 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.170 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.170 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.170 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.170 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.170 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:10:24.171 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.171 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.171 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.171 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.171 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.171 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.171 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.171 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:10:24.171 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.568985+0000" 2024-09-22T03:10:24.171 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.171 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.171 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.171 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.171 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.171 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.172 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:10:24.172 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.172 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:10:24.172 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:10:24.172 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.172 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-22T03:10:24.172 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "client_request", 2024-09-22T03:10:24.172 INFO:teuthology.orchestra.run.smithi153.stdout: "id": 598, 2024-09-22T03:10:24.172 INFO:teuthology.orchestra.run.smithi153.stdout: "detail": { 2024-09-22T03:10:24.172 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.172 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:10:24.172 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.612538+0000" 2024-09-22T03:10:24.172 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.172 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.172 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:10:24.173 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.612550+0000" 2024-09-22T03:10:24.173 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.173 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.173 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.173 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.173 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.173 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.173 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:10:24.173 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.612741+0000" 2024-09-22T03:10:24.173 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.173 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.173 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.173 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.173 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.173 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.174 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:10:24.174 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.174 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.174 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.174 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.174 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.174 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.174 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.174 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:10:24.174 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.612826+0000" 2024-09-22T03:10:24.174 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.174 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.174 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.174 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.175 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.175 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.175 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:10:24.175 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.613076+0000" 2024-09-22T03:10:24.175 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.175 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.175 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.175 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.175 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.175 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.175 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:10:24.175 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.175 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.175 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.175 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.176 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.176 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.176 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.176 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:10:24.176 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.623952+0000" 2024-09-22T03:10:24.176 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.176 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.176 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:10:24.176 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.613469+0000" 2024-09-22T03:10:24.176 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.176 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.176 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.176 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.176 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.176 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.177 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:10:24.177 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.177 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.177 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.177 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.177 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.177 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.177 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.177 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:10:24.177 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.613632+0000" 2024-09-22T03:10:24.177 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.177 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.177 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.177 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.178 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.178 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.178 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:10:24.178 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.178 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.178 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.178 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.178 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.178 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.178 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.178 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:10:24.178 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.613739+0000" 2024-09-22T03:10:24.178 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.178 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.179 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.179 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.179 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.179 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.179 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:10:24.179 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.179 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.179 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.179 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.179 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.179 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.179 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.179 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:10:24.179 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.179 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.180 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.180 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.180 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.180 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.180 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.180 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:10:24.180 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.180 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.180 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.180 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.180 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.180 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.180 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.180 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:10:24.180 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.614039+0000" 2024-09-22T03:10:24.181 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.181 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.181 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.181 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.181 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.181 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.181 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:10:24.181 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.181 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.181 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.181 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.181 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.181 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.181 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.182 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:10:24.182 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.614780+0000" 2024-09-22T03:10:24.182 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.182 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.182 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.182 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.182 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.182 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.182 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:10:24.182 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.182 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.182 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.182 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.182 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.182 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.183 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.183 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:10:24.183 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.623251+0000" 2024-09-22T03:10:24.183 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.183 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.183 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.183 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.183 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.183 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.183 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:10:24.183 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.183 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:10:24.183 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:10:24.183 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.183 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-22T03:10:24.184 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "client_request", 2024-09-22T03:10:24.184 INFO:teuthology.orchestra.run.smithi153.stdout: "id": 603, 2024-09-22T03:10:24.184 INFO:teuthology.orchestra.run.smithi153.stdout: "detail": { 2024-09-22T03:10:24.184 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.184 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:10:24.184 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.625223+0000" 2024-09-22T03:10:24.184 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.184 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.184 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:10:24.184 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.625235+0000" 2024-09-22T03:10:24.184 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.184 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.184 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.184 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.184 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.185 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.185 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:10:24.185 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.625415+0000" 2024-09-22T03:10:24.185 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.185 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.185 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.185 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.185 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.185 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.185 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:10:24.185 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.185 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.185 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.185 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.185 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.186 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.186 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.186 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:10:24.186 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.625503+0000" 2024-09-22T03:10:24.186 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.186 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.186 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.186 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.186 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.186 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.186 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:10:24.186 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.626627+0000" 2024-09-22T03:10:24.186 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.186 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.187 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.187 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.187 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.187 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.187 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:10:24.187 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.187 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.187 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.187 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.187 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.187 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.187 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.187 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:10:24.187 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.641721+0000" 2024-09-22T03:10:24.187 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.188 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.188 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:10:24.188 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.627046+0000" 2024-09-22T03:10:24.188 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.188 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.188 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.188 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.188 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.188 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.188 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:10:24.188 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.188 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.188 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.188 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.189 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.189 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.189 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.189 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:10:24.189 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.627229+0000" 2024-09-22T03:10:24.189 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.189 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.189 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.189 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.189 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.189 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.189 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:10:24.189 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.189 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.189 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.190 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.190 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.190 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.190 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.190 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:10:24.190 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.627355+0000" 2024-09-22T03:10:24.190 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.190 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.190 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.190 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.190 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.190 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.190 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:10:24.191 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.191 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.191 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.191 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.191 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.191 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.191 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.191 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:10:24.191 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.191 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.191 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.191 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.191 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.191 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.192 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.192 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:10:24.192 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.192 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.192 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.192 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.192 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.192 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.192 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.192 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:10:24.192 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.627560+0000" 2024-09-22T03:10:24.192 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.192 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.192 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.193 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.193 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.193 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.193 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:10:24.193 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.193 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.193 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.193 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.193 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.193 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.193 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.193 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:10:24.193 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.628065+0000" 2024-09-22T03:10:24.193 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.194 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.194 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.194 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.194 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.194 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.194 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:10:24.194 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.194 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.194 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.194 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.194 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.194 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.194 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.194 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:10:24.195 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.641299+0000" 2024-09-22T03:10:24.195 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.195 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.195 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.195 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.195 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.195 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.195 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:10:24.195 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.195 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:10:24.195 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:10:24.195 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.195 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-22T03:10:24.195 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "client_request", 2024-09-22T03:10:24.196 INFO:teuthology.orchestra.run.smithi153.stdout: "id": 608, 2024-09-22T03:10:24.196 INFO:teuthology.orchestra.run.smithi153.stdout: "detail": { 2024-09-22T03:10:24.196 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.196 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:10:24.196 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.642579+0000" 2024-09-22T03:10:24.196 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.196 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.196 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:10:24.196 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.642588+0000" 2024-09-22T03:10:24.196 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.196 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.196 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.196 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.196 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.197 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.197 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:10:24.197 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.642704+0000" 2024-09-22T03:10:24.197 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.197 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.197 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.197 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.197 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.197 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.197 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:10:24.197 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.197 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.197 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.197 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.197 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.198 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.198 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.198 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:10:24.198 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.642817+0000" 2024-09-22T03:10:24.198 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.198 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.198 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.198 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.198 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.198 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.198 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:10:24.198 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.643034+0000" 2024-09-22T03:10:24.198 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.198 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.199 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.199 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.199 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.199 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.199 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:10:24.199 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.199 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.199 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.199 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.199 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.199 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.199 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.199 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:10:24.199 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.657076+0000" 2024-09-22T03:10:24.200 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.200 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.200 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:10:24.200 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.643445+0000" 2024-09-22T03:10:24.200 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.200 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.200 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.200 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.200 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.200 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.200 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:10:24.200 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.200 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.200 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.201 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.201 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.201 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.201 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.201 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:10:24.201 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.643631+0000" 2024-09-22T03:10:24.201 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.201 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.201 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.201 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.201 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.201 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.201 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:10:24.201 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.201 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.202 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.202 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.202 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.202 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.202 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.202 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:10:24.202 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.643830+0000" 2024-09-22T03:10:24.202 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.202 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.202 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.202 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.202 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.202 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.202 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:10:24.203 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.203 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.203 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.203 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.203 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.203 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.203 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.203 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:10:24.203 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.203 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.203 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.203 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.203 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.203 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.204 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.204 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:10:24.204 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.204 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.204 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.204 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.204 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.204 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.204 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.204 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:10:24.204 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.644047+0000" 2024-09-22T03:10:24.204 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.204 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.204 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.204 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.205 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.205 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.205 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:10:24.205 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.205 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.205 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.205 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.205 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.205 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.205 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.205 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:10:24.205 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.644325+0000" 2024-09-22T03:10:24.205 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.205 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.205 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.206 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.206 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.206 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.206 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:10:24.206 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.206 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.206 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.206 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.206 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.206 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.206 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.206 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:10:24.206 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.656782+0000" 2024-09-22T03:10:24.206 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.206 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.207 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.207 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.207 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.207 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.207 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:10:24.207 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.207 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:10:24.207 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:10:24.207 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.207 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-22T03:10:24.207 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "client_request", 2024-09-22T03:10:24.207 INFO:teuthology.orchestra.run.smithi153.stdout: "id": 609, 2024-09-22T03:10:24.207 INFO:teuthology.orchestra.run.smithi153.stdout: "detail": { 2024-09-22T03:10:24.207 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.207 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:10:24.208 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.643554+0000" 2024-09-22T03:10:24.208 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.208 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.208 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:10:24.208 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.643562+0000" 2024-09-22T03:10:24.208 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.208 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.208 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.208 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.208 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.208 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.208 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:10:24.208 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.643655+0000" 2024-09-22T03:10:24.208 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.208 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.209 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.209 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.209 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.209 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.209 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:10:24.209 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.209 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.209 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.209 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.209 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.209 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.209 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.209 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:10:24.209 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.643717+0000" 2024-09-22T03:10:24.209 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.210 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.210 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.210 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.210 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.210 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.210 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:10:24.210 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.646268+0000" 2024-09-22T03:10:24.210 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.210 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.210 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.210 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.210 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.210 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.210 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:10:24.211 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.211 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.211 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.211 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.211 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.211 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.211 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.211 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:10:24.211 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.657366+0000" 2024-09-22T03:10:24.211 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.211 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.211 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:10:24.211 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.646729+0000" 2024-09-22T03:10:24.211 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.211 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.212 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.212 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.212 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.212 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.212 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:10:24.212 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.212 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.212 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.212 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.212 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.212 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.212 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.212 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:10:24.212 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.646877+0000" 2024-09-22T03:10:24.213 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.213 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.213 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.213 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.213 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.213 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.213 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:10:24.213 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.213 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.213 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.213 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.213 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.213 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.213 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.213 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:10:24.214 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.647211+0000" 2024-09-22T03:10:24.214 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.214 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.214 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.214 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.214 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.214 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.214 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:10:24.214 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.214 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.214 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.214 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.214 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.214 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.215 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.215 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:10:24.215 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.215 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.215 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.215 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.215 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.215 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.215 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.215 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:10:24.215 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.215 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.215 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.215 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.215 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.216 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.216 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.216 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:10:24.216 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.647742+0000" 2024-09-22T03:10:24.216 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.216 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.216 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.216 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.216 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.216 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.216 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:10:24.216 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.216 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.216 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.217 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.217 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.217 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.217 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.217 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:10:24.217 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.648125+0000" 2024-09-22T03:10:24.217 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.217 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.217 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.217 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.217 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.217 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.217 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:10:24.217 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.217 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.218 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.218 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.218 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.656689+0000" 2024-09-22T03:10:24.218 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.218 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.218 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:10:24.218 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.656895+0000" 2024-09-22T03:10:24.218 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.218 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.218 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.218 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.218 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.218 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.218 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:10:24.219 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.219 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:10:24.219 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:10:24.219 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.219 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-22T03:10:24.219 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "client_request", 2024-09-22T03:10:24.219 INFO:teuthology.orchestra.run.smithi153.stdout: "id": 610, 2024-09-22T03:10:24.219 INFO:teuthology.orchestra.run.smithi153.stdout: "detail": { 2024-09-22T03:10:24.219 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.219 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:10:24.219 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.644066+0000" 2024-09-22T03:10:24.219 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.219 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.219 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:10:24.220 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.644070+0000" 2024-09-22T03:10:24.220 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.220 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.220 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.220 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.220 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.220 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.220 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:10:24.220 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.644139+0000" 2024-09-22T03:10:24.220 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.220 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.220 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.220 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.220 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.220 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.221 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:10:24.221 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.221 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.221 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.221 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.221 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.221 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.221 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.221 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:10:24.221 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.644186+0000" 2024-09-22T03:10:24.221 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.221 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.221 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.221 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.222 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.222 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.222 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:10:24.222 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.646416+0000" 2024-09-22T03:10:24.222 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.222 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.222 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.222 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.222 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.222 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.222 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:10:24.222 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.222 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.222 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.222 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.223 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.223 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.223 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.223 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:10:24.223 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.662251+0000" 2024-09-22T03:10:24.223 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.223 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.223 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:10:24.223 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.647097+0000" 2024-09-22T03:10:24.223 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.223 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.223 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.223 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.223 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.223 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.224 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:10:24.224 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.224 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.224 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.224 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.224 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.224 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.224 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.224 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:10:24.224 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.647317+0000" 2024-09-22T03:10:24.224 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.224 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.224 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.224 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.224 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.225 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.225 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:10:24.225 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.225 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.225 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.225 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.225 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.225 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.225 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.225 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:10:24.225 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.647691+0000" 2024-09-22T03:10:24.225 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.225 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.225 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.226 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.226 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.226 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.226 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:10:24.226 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.226 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.226 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.226 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.226 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.226 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.226 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.226 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:10:24.226 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.226 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.226 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.227 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.227 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.227 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.227 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.227 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:10:24.227 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.227 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.227 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.227 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.227 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.227 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.227 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.227 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:10:24.227 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.648805+0000" 2024-09-22T03:10:24.227 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.228 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.228 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.228 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.228 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.228 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.228 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:10:24.228 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.228 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.228 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.228 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.228 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.228 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.228 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.228 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:10:24.228 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.650840+0000" 2024-09-22T03:10:24.229 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.229 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.229 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.229 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.229 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.229 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.229 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:10:24.229 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.229 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.229 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.229 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.229 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.229 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.229 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.229 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:10:24.230 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.661716+0000" 2024-09-22T03:10:24.230 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.230 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.230 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.230 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.230 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.230 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.230 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:10:24.230 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.230 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:10:24.230 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:10:24.230 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.230 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-22T03:10:24.230 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "client_request", 2024-09-22T03:10:24.231 INFO:teuthology.orchestra.run.smithi153.stdout: "id": 611, 2024-09-22T03:10:24.231 INFO:teuthology.orchestra.run.smithi153.stdout: "detail": { 2024-09-22T03:10:24.231 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.231 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:10:24.231 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.644335+0000" 2024-09-22T03:10:24.231 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.231 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.231 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:10:24.231 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.644338+0000" 2024-09-22T03:10:24.231 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.231 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.231 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.231 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.231 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.232 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.232 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:10:24.232 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.644399+0000" 2024-09-22T03:10:24.232 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.232 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.232 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.232 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.232 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.232 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.232 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:10:24.232 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.232 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.232 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.232 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.232 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.233 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.233 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.233 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:10:24.233 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.644448+0000" 2024-09-22T03:10:24.233 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.233 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.233 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.233 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.233 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.233 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.233 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:10:24.233 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.646446+0000" 2024-09-22T03:10:24.233 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.233 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.234 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.234 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.234 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.234 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.234 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:10:24.234 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.234 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.234 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.234 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.234 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.234 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.234 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.234 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:10:24.234 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.669017+0000" 2024-09-22T03:10:24.234 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.235 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.235 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:10:24.235 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.647483+0000" 2024-09-22T03:10:24.235 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.235 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.235 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.235 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.235 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.235 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.235 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:10:24.235 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.235 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.235 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.235 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.236 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.236 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.236 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.236 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:10:24.236 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.647827+0000" 2024-09-22T03:10:24.236 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.236 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.236 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.236 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.236 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.236 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.236 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:10:24.236 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.236 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.236 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.237 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.237 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.237 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.237 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.237 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:10:24.237 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.649071+0000" 2024-09-22T03:10:24.237 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.237 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.237 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.237 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.237 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.237 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.237 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:10:24.237 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.237 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.238 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.238 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.238 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.238 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.238 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.238 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:10:24.238 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.238 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.238 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.238 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.238 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.238 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.238 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.238 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:10:24.239 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.239 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.239 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.239 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.239 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.239 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.239 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.239 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:10:24.239 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.650552+0000" 2024-09-22T03:10:24.239 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.239 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.239 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.239 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.239 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.240 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.240 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:10:24.240 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.240 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.240 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.240 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.240 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.651320+0000" 2024-09-22T03:10:24.240 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.240 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.240 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:10:24.240 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.652709+0000" 2024-09-22T03:10:24.240 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.240 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.240 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.240 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.241 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.241 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.241 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:10:24.241 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.241 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.241 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.241 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.241 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.241 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.241 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.241 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:10:24.241 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.668097+0000" 2024-09-22T03:10:24.241 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.241 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.241 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.242 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.242 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.242 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.242 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:10:24.242 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.242 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:10:24.242 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:10:24.242 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.242 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-22T03:10:24.242 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "client_request", 2024-09-22T03:10:24.242 INFO:teuthology.orchestra.run.smithi153.stdout: "id": 612, 2024-09-22T03:10:24.242 INFO:teuthology.orchestra.run.smithi153.stdout: "detail": { 2024-09-22T03:10:24.242 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.242 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:10:24.243 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.644669+0000" 2024-09-22T03:10:24.243 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.243 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.243 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:10:24.243 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.644673+0000" 2024-09-22T03:10:24.243 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.243 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.243 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.243 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.243 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.243 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.243 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:10:24.243 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.644731+0000" 2024-09-22T03:10:24.243 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.243 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.244 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.244 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.244 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.244 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.244 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:10:24.244 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.244 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.244 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.244 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.244 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.244 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.244 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.244 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:10:24.244 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.644774+0000" 2024-09-22T03:10:24.245 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.245 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.245 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.245 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.245 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.245 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.245 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:10:24.245 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.646502+0000" 2024-09-22T03:10:24.245 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.245 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.245 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.245 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.245 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.245 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.246 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:10:24.246 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.246 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.246 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.246 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.246 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.246 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.246 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.246 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:10:24.246 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.672183+0000" 2024-09-22T03:10:24.246 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.246 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.246 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:10:24.246 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.648303+0000" 2024-09-22T03:10:24.247 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.247 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.247 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.247 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.247 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.247 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.247 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:10:24.247 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.247 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.247 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.247 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.247 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.247 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.247 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.247 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:10:24.248 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.650453+0000" 2024-09-22T03:10:24.248 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.248 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.248 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.248 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.248 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.248 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.248 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:10:24.248 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.248 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.248 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.248 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.248 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.248 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.249 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.249 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:10:24.249 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.651059+0000" 2024-09-22T03:10:24.249 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.249 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.249 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.249 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.249 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.249 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.249 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:10:24.249 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.249 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.249 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.249 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.249 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.250 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.250 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.250 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:10:24.250 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.250 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.250 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.250 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.250 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.250 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.250 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.250 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:10:24.250 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.250 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.250 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.250 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.251 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.251 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.251 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.251 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:10:24.251 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.651449+0000" 2024-09-22T03:10:24.251 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.251 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.251 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.251 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.251 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.251 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.251 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:10:24.251 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.251 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.251 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.252 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.252 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.651815+0000" 2024-09-22T03:10:24.252 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.252 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.252 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:10:24.252 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.654775+0000" 2024-09-22T03:10:24.252 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.252 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.252 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.252 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.252 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.252 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.252 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:10:24.252 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.252 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.253 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.253 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.253 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.253 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.253 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.253 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:10:24.253 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.670887+0000" 2024-09-22T03:10:24.253 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.253 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.253 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.253 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.253 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.253 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.253 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:10:24.254 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.254 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:10:24.254 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:10:24.254 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.254 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-22T03:10:24.254 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "client_request", 2024-09-22T03:10:24.254 INFO:teuthology.orchestra.run.smithi153.stdout: "id": 613, 2024-09-22T03:10:24.254 INFO:teuthology.orchestra.run.smithi153.stdout: "detail": { 2024-09-22T03:10:24.254 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.254 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:10:24.254 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.644992+0000" 2024-09-22T03:10:24.254 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.254 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.254 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:10:24.255 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.644995+0000" 2024-09-22T03:10:24.255 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.255 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.255 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.255 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.255 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.255 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.255 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:10:24.255 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.645058+0000" 2024-09-22T03:10:24.255 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.255 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.255 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.255 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.255 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.256 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.256 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:10:24.256 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.256 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.256 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.256 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.256 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.256 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.256 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.256 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:10:24.256 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.645105+0000" 2024-09-22T03:10:24.256 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.256 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.256 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.256 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.257 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.257 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.257 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:10:24.257 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.646921+0000" 2024-09-22T03:10:24.257 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.257 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.257 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.257 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.257 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.257 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.257 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:10:24.257 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.257 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.257 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.257 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.258 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.258 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.258 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.258 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:10:24.258 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.678037+0000" 2024-09-22T03:10:24.258 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.258 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.258 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:10:24.258 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.650749+0000" 2024-09-22T03:10:24.258 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.258 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.258 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.258 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.258 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.259 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.259 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:10:24.259 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.259 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.259 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.259 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.259 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.259 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.259 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.259 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:10:24.259 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.650962+0000" 2024-09-22T03:10:24.259 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.259 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.259 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.260 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.260 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.260 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.260 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:10:24.260 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.260 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.260 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.260 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.260 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.260 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.260 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.260 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:10:24.260 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.651582+0000" 2024-09-22T03:10:24.260 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.260 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.261 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.261 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.261 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.261 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.261 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:10:24.261 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.261 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.261 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.261 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.261 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.261 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.261 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.261 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:10:24.261 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.261 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.262 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.262 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.262 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.262 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.262 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.262 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:10:24.262 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.262 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.262 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.262 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.262 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.262 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.262 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.262 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:10:24.263 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.652027+0000" 2024-09-22T03:10:24.263 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.263 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.263 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.263 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.263 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.263 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.263 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:10:24.263 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.263 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.263 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.263 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.263 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.652439+0000" 2024-09-22T03:10:24.263 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.263 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.264 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:10:24.264 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.658438+0000" 2024-09-22T03:10:24.264 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.264 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.264 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.264 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.264 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.264 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.264 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:10:24.264 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.264 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.264 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.264 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.264 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.265 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.265 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.265 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:10:24.265 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.673294+0000" 2024-09-22T03:10:24.265 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.265 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.265 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.265 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.265 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.265 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.265 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:10:24.265 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.265 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:10:24.265 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:10:24.266 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.266 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-22T03:10:24.266 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "client_request", 2024-09-22T03:10:24.266 INFO:teuthology.orchestra.run.smithi153.stdout: "id": 614, 2024-09-22T03:10:24.266 INFO:teuthology.orchestra.run.smithi153.stdout: "detail": { 2024-09-22T03:10:24.266 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.266 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:10:24.266 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.659780+0000" 2024-09-22T03:10:24.266 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.266 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.266 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:10:24.266 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.659790+0000" 2024-09-22T03:10:24.266 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.266 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.266 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.267 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.267 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.267 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.267 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:10:24.267 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.659898+0000" 2024-09-22T03:10:24.267 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.267 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.267 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.267 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.267 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.267 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.267 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:10:24.267 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.267 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.268 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.268 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.268 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.268 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.268 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.268 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:10:24.268 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.659954+0000" 2024-09-22T03:10:24.268 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.268 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.268 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.268 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.268 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.268 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.268 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:10:24.268 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.660535+0000" 2024-09-22T03:10:24.269 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.269 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.269 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.269 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.269 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.269 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.269 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:10:24.269 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.269 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.269 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.269 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.269 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.269 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.269 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.270 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:10:24.270 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.680795+0000" 2024-09-22T03:10:24.270 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.270 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.270 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:10:24.270 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.660779+0000" 2024-09-22T03:10:24.270 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.270 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.270 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.270 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.270 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.270 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.270 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:10:24.270 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.271 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.271 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.271 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.271 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.271 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.271 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.271 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:10:24.271 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.660925+0000" 2024-09-22T03:10:24.271 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.271 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.271 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.271 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.271 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.271 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.271 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:10:24.272 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.272 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.272 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.272 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.272 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.272 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.272 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.272 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:10:24.272 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.661039+0000" 2024-09-22T03:10:24.272 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.272 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.272 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.272 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.272 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.272 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.273 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:10:24.273 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.273 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.273 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.273 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.273 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.273 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.273 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.273 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:10:24.273 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.273 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.273 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.273 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.273 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.273 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.274 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.274 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:10:24.274 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.274 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.274 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.274 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.274 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.274 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.274 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.274 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:10:24.274 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.661209+0000" 2024-09-22T03:10:24.274 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.274 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.274 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.274 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.275 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.275 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.275 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:10:24.275 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.275 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.275 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.275 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.275 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.275 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.275 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.275 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:10:24.275 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.664487+0000" 2024-09-22T03:10:24.275 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.275 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.276 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.276 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.276 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.276 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.276 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:10:24.276 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.276 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.276 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.276 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.276 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.276 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.276 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.276 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:10:24.276 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.679276+0000" 2024-09-22T03:10:24.277 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.277 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.277 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.277 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.277 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.277 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.277 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:10:24.277 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.277 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:10:24.277 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:10:24.277 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.277 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-22T03:10:24.277 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "client_request", 2024-09-22T03:10:24.277 INFO:teuthology.orchestra.run.smithi153.stdout: "id": 615, 2024-09-22T03:10:24.277 INFO:teuthology.orchestra.run.smithi153.stdout: "detail": { 2024-09-22T03:10:24.278 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.278 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:10:24.278 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.661523+0000" 2024-09-22T03:10:24.278 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.278 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.278 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:10:24.278 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.661533+0000" 2024-09-22T03:10:24.278 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.278 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.278 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.278 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.278 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.278 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.278 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:10:24.279 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.662112+0000" 2024-09-22T03:10:24.279 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.279 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.279 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.279 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.279 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.279 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.279 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:10:24.279 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.279 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.279 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.279 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.279 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.279 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.279 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.280 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:10:24.280 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.662201+0000" 2024-09-22T03:10:24.280 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.280 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.280 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.280 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.280 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.280 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.280 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:10:24.280 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.663720+0000" 2024-09-22T03:10:24.280 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.280 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.280 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.280 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.281 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.281 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.281 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:10:24.281 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.281 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.281 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.281 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.281 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.281 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.281 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.281 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:10:24.281 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.688911+0000" 2024-09-22T03:10:24.281 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.281 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.281 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:10:24.282 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.664021+0000" 2024-09-22T03:10:24.282 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.282 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.282 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.282 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.282 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.282 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.282 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:10:24.282 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.282 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.282 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.282 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.282 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.282 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.283 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.283 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:10:24.283 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.664468+0000" 2024-09-22T03:10:24.283 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.283 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.283 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.283 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.283 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.283 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.283 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:10:24.283 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.283 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.283 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.283 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.283 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.284 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.284 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.284 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:10:24.284 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.664806+0000" 2024-09-22T03:10:24.284 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.284 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.284 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.284 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.284 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.284 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.284 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:10:24.284 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.284 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.284 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.284 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.285 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.285 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.285 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.285 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:10:24.285 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.285 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.285 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.285 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.285 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.285 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.285 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.285 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:10:24.285 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.285 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.285 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.286 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.286 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.286 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.286 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.286 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:10:24.286 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.666726+0000" 2024-09-22T03:10:24.286 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.286 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.286 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.286 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.286 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.286 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.286 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:10:24.287 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.287 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.287 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.287 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.287 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.287 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.287 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.287 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:10:24.287 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.668920+0000" 2024-09-22T03:10:24.287 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.287 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.287 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.287 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.287 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.288 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.288 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:10:24.288 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.288 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.288 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.288 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.288 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.288 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.288 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.288 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:10:24.288 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.688363+0000" 2024-09-22T03:10:24.288 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.288 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.288 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.289 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.289 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.289 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.289 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:10:24.289 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.289 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:10:24.289 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:10:24.289 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.289 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-22T03:10:24.289 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "client_request", 2024-09-22T03:10:24.289 INFO:teuthology.orchestra.run.smithi153.stdout: "id": 616, 2024-09-22T03:10:24.289 INFO:teuthology.orchestra.run.smithi153.stdout: "detail": { 2024-09-22T03:10:24.289 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.289 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:10:24.289 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.662809+0000" 2024-09-22T03:10:24.290 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.290 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.290 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:10:24.290 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.662815+0000" 2024-09-22T03:10:24.290 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.290 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.290 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.290 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.290 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.290 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.290 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:10:24.290 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.662959+0000" 2024-09-22T03:10:24.290 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.291 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.291 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.291 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.291 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.291 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.291 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:10:24.291 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.291 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.291 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.291 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.291 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.291 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.291 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.291 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:10:24.291 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.663037+0000" 2024-09-22T03:10:24.292 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.292 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.292 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.292 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.292 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.292 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.292 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:10:24.292 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.663756+0000" 2024-09-22T03:10:24.292 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.292 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.292 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.292 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.292 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.292 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.293 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:10:24.293 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.293 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.293 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.293 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.293 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.293 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.293 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.293 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:10:24.293 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.701116+0000" 2024-09-22T03:10:24.293 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.293 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.293 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:10:24.293 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.664703+0000" 2024-09-22T03:10:24.293 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.294 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.294 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.294 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.294 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.294 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.294 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:10:24.294 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.294 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.294 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.294 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.294 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.294 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.294 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.294 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:10:24.294 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.665859+0000" 2024-09-22T03:10:24.295 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.295 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.295 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.295 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.295 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.295 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.295 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:10:24.295 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.295 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.295 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.295 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.295 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.295 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.295 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.295 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:10:24.296 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.666778+0000" 2024-09-22T03:10:24.296 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.296 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.296 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.296 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.296 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.296 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.296 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:10:24.296 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.296 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.296 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.296 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.296 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.296 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.297 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.297 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:10:24.297 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.297 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.297 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.297 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.297 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.297 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.297 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.297 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:10:24.297 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.297 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.297 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.297 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.297 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.298 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.298 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.298 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:10:24.298 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.669212+0000" 2024-09-22T03:10:24.298 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.298 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.298 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.298 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.298 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.298 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.298 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:10:24.298 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.298 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.298 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.299 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.299 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.669782+0000" 2024-09-22T03:10:24.299 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.299 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.299 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:10:24.299 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.671534+0000" 2024-09-22T03:10:24.299 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.299 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.299 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.299 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.299 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.299 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.299 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:10:24.299 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.299 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.300 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.300 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.300 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.300 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.300 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.300 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:10:24.300 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.700698+0000" 2024-09-22T03:10:24.300 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.300 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.300 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.300 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.300 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.300 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.300 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:10:24.300 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.301 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:10:24.301 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:10:24.301 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.301 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-22T03:10:24.301 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "client_request", 2024-09-22T03:10:24.301 INFO:teuthology.orchestra.run.smithi153.stdout: "id": 617, 2024-09-22T03:10:24.301 INFO:teuthology.orchestra.run.smithi153.stdout: "detail": { 2024-09-22T03:10:24.301 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.301 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:10:24.301 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.664502+0000" 2024-09-22T03:10:24.301 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.301 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.301 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:10:24.301 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.664511+0000" 2024-09-22T03:10:24.302 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.302 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.302 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.302 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.302 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.302 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.302 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:10:24.302 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.664776+0000" 2024-09-22T03:10:24.302 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.302 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.302 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.302 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.302 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.302 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.302 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:10:24.303 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.303 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.303 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.303 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.303 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.303 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.303 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.303 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:10:24.303 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.665017+0000" 2024-09-22T03:10:24.303 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.303 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.303 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.303 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.303 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.304 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.304 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:10:24.304 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.665930+0000" 2024-09-22T03:10:24.304 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.304 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.304 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.304 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.304 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.304 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.304 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:10:24.304 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.304 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.304 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.304 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.304 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.305 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.305 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.305 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:10:24.305 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.710470+0000" 2024-09-22T03:10:24.305 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.305 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.305 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:10:24.305 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.666507+0000" 2024-09-22T03:10:24.305 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.305 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.305 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.305 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.305 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.305 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.305 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:10:24.306 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.306 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.306 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.306 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.306 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.306 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.306 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.306 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:10:24.306 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.668088+0000" 2024-09-22T03:10:24.306 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.306 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.306 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.306 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.306 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.307 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.307 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:10:24.307 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.307 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.307 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.307 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.307 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.307 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.307 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.307 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:10:24.307 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.669981+0000" 2024-09-22T03:10:24.307 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.307 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.307 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.308 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.308 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.308 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.308 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:10:24.308 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.308 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.308 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.308 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.308 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.308 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.308 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.308 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:10:24.308 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.308 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.308 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.309 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.309 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.309 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.309 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.309 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:10:24.309 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.309 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.309 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.309 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.309 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.309 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.309 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.309 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:10:24.309 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.670787+0000" 2024-09-22T03:10:24.309 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.310 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.310 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.310 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.310 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.310 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.310 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:10:24.310 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.310 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.310 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.310 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.310 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.671302+0000" 2024-09-22T03:10:24.310 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.310 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.310 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:10:24.311 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.680987+0000" 2024-09-22T03:10:24.311 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.311 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.311 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.311 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.311 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.311 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.311 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:10:24.311 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.311 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.311 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.311 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.311 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.311 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.312 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.312 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:10:24.312 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.710006+0000" 2024-09-22T03:10:24.312 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.312 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.312 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.312 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.312 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.312 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.312 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:10:24.312 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.312 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:10:24.312 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:10:24.312 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.313 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-22T03:10:24.313 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "client_request", 2024-09-22T03:10:24.313 INFO:teuthology.orchestra.run.smithi153.stdout: "id": 618, 2024-09-22T03:10:24.313 INFO:teuthology.orchestra.run.smithi153.stdout: "detail": { 2024-09-22T03:10:24.313 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.313 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:10:24.313 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.667258+0000" 2024-09-22T03:10:24.313 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.313 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.313 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:10:24.313 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.667265+0000" 2024-09-22T03:10:24.313 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.313 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.313 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.313 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.314 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.314 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.314 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:10:24.314 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.667379+0000" 2024-09-22T03:10:24.314 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.314 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.314 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.314 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.314 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.314 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.314 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:10:24.314 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.314 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.314 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.315 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.315 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.315 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.315 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.315 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:10:24.315 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.667450+0000" 2024-09-22T03:10:24.315 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.315 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.315 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.315 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.315 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.316 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.316 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:10:24.316 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.668783+0000" 2024-09-22T03:10:24.316 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.316 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.316 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.316 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.316 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.316 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.316 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:10:24.316 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.317 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.317 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.317 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.317 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.317 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.317 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.317 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:10:24.317 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.712195+0000" 2024-09-22T03:10:24.317 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.317 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.317 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:10:24.317 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.669487+0000" 2024-09-22T03:10:24.317 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.318 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.318 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.318 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.318 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.318 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.318 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:10:24.318 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.318 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.318 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.318 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.318 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.319 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.319 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.319 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:10:24.319 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.670193+0000" 2024-09-22T03:10:24.319 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.319 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.319 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.319 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.319 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.319 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.319 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:10:24.320 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.320 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.320 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.320 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.320 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.320 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.320 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.320 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:10:24.320 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.671920+0000" 2024-09-22T03:10:24.320 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.320 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.320 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.320 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.321 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.321 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.321 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:10:24.321 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.321 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.321 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.321 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.321 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.321 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.321 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.322 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:10:24.322 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.322 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.322 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.322 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.322 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.322 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.322 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.322 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:10:24.322 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.322 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.322 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.323 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.323 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.323 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.323 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.323 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:10:24.323 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.675620+0000" 2024-09-22T03:10:24.323 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.323 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.323 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.323 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.323 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.324 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.324 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:10:24.324 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.324 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.324 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.324 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.324 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.676576+0000" 2024-09-22T03:10:24.324 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.324 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.324 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:10:24.324 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.689209+0000" 2024-09-22T03:10:24.324 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.325 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.325 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.325 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.325 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.325 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.325 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:10:24.325 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.325 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.325 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.325 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.325 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.325 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.326 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.326 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:10:24.326 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.710976+0000" 2024-09-22T03:10:24.326 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.326 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.326 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.326 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.326 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.326 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.326 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:10:24.326 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.326 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:10:24.327 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:10:24.327 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.327 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-22T03:10:24.327 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "client_request", 2024-09-22T03:10:24.327 INFO:teuthology.orchestra.run.smithi153.stdout: "id": 619, 2024-09-22T03:10:24.327 INFO:teuthology.orchestra.run.smithi153.stdout: "detail": { 2024-09-22T03:10:24.327 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.327 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:10:24.327 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.671200+0000" 2024-09-22T03:10:24.327 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.327 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.328 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:10:24.328 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.671210+0000" 2024-09-22T03:10:24.328 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.328 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.328 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.328 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.328 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.328 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.328 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:10:24.328 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.671297+0000" 2024-09-22T03:10:24.328 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.328 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.329 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.329 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.329 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.329 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.329 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:10:24.329 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.329 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.329 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.329 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.329 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.329 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.330 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.330 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:10:24.330 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.671366+0000" 2024-09-22T03:10:24.330 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.330 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.330 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.330 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.330 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.330 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.330 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:10:24.330 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.672538+0000" 2024-09-22T03:10:24.330 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.331 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.331 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.331 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.331 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.331 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.331 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:10:24.331 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.331 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.331 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.331 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.331 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.332 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.332 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.332 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:10:24.332 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.719819+0000" 2024-09-22T03:10:24.332 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.332 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.332 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:10:24.332 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.675466+0000" 2024-09-22T03:10:24.332 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.332 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.332 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.332 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.333 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.333 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.333 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:10:24.333 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.333 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.333 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.333 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.333 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.333 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.333 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.333 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:10:24.334 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.676019+0000" 2024-09-22T03:10:24.334 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.334 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.334 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.334 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.334 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.334 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.334 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:10:24.334 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.334 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.334 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.335 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.335 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.335 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.335 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.335 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:10:24.335 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.676707+0000" 2024-09-22T03:10:24.335 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.335 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.335 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.335 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.335 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.335 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.336 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:10:24.336 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.336 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.336 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.336 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.336 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.336 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.336 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.336 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:10:24.336 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.336 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.336 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.337 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.337 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.337 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.337 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.337 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:10:24.337 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.337 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.337 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.337 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.337 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.337 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.337 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.338 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:10:24.338 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.677558+0000" 2024-09-22T03:10:24.338 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.338 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.338 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.338 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.338 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.338 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.338 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:10:24.338 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.338 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.338 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.339 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.339 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.679387+0000" 2024-09-22T03:10:24.339 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.339 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.339 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:10:24.339 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.694450+0000" 2024-09-22T03:10:24.339 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.339 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.339 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.339 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.339 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.339 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.339 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:10:24.340 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.340 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.340 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.340 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.340 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.340 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.340 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.340 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:10:24.340 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.719438+0000" 2024-09-22T03:10:24.340 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.340 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.340 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.341 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.341 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.341 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.341 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:10:24.341 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.341 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:10:24.341 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:10:24.341 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.341 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-22T03:10:24.341 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "client_request", 2024-09-22T03:10:24.341 INFO:teuthology.orchestra.run.smithi153.stdout: "id": 620, 2024-09-22T03:10:24.341 INFO:teuthology.orchestra.run.smithi153.stdout: "detail": { 2024-09-22T03:10:24.342 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.342 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:10:24.342 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.671618+0000" 2024-09-22T03:10:24.342 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.342 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.342 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:10:24.342 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.671621+0000" 2024-09-22T03:10:24.342 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.342 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.342 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.342 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.342 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.343 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.343 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:10:24.343 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.671686+0000" 2024-09-22T03:10:24.343 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.343 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.343 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.343 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.343 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.343 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.343 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:10:24.343 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.343 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.344 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.344 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.344 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.344 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.344 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.344 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:10:24.344 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.671740+0000" 2024-09-22T03:10:24.344 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.344 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.344 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.344 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.344 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.345 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.345 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:10:24.345 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.675083+0000" 2024-09-22T03:10:24.345 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.345 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.345 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.345 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.345 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.345 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.345 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:10:24.345 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.345 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.345 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.346 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.346 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.346 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.346 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.346 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:10:24.346 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.721231+0000" 2024-09-22T03:10:24.346 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.346 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.346 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:10:24.347 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.677201+0000" 2024-09-22T03:10:24.347 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.347 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.347 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.347 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.347 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.347 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.347 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:10:24.347 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.347 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.347 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.347 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.348 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.348 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.348 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.348 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:10:24.348 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.678267+0000" 2024-09-22T03:10:24.348 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.348 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.348 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.348 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.348 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.348 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.348 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:10:24.348 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.349 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.349 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.349 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.349 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.349 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.349 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.349 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:10:24.349 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.678849+0000" 2024-09-22T03:10:24.349 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.349 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.349 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.349 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.350 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.350 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.350 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:10:24.350 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.350 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.350 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.350 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.350 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.350 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.350 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.350 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:10:24.350 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.351 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.351 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.351 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.351 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.351 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.351 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.351 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:10:24.351 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.351 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.351 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.351 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.351 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.352 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.352 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.352 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:10:24.352 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.679844+0000" 2024-09-22T03:10:24.352 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.352 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.352 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.352 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.352 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.352 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.352 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:10:24.353 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.353 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.353 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.353 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.353 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.680520+0000" 2024-09-22T03:10:24.353 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.353 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.353 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:10:24.353 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.698105+0000" 2024-09-22T03:10:24.353 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.353 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.353 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.353 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.354 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.354 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.354 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:10:24.354 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.354 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.354 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.354 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.354 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.354 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.354 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.354 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:10:24.355 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.720861+0000" 2024-09-22T03:10:24.355 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.355 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.355 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.355 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.355 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.355 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.355 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:10:24.355 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.355 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:10:24.355 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:10:24.355 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.355 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-22T03:10:24.356 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "client_request", 2024-09-22T03:10:24.356 INFO:teuthology.orchestra.run.smithi153.stdout: "id": 621, 2024-09-22T03:10:24.356 INFO:teuthology.orchestra.run.smithi153.stdout: "detail": { 2024-09-22T03:10:24.356 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.357 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:10:24.357 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.672949+0000" 2024-09-22T03:10:24.357 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.357 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.357 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:10:24.358 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.672956+0000" 2024-09-22T03:10:24.358 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.358 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.358 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.358 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.358 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.358 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.358 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:10:24.358 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.673117+0000" 2024-09-22T03:10:24.358 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.358 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.359 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.359 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.359 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.359 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.359 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:10:24.359 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.359 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.359 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.359 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.359 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.359 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.359 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.360 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:10:24.360 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.673197+0000" 2024-09-22T03:10:24.360 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.360 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.360 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.360 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.360 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.360 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.360 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:10:24.360 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.675802+0000" 2024-09-22T03:10:24.360 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.360 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.360 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.361 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.361 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.361 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.361 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:10:24.361 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.361 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.361 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.361 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.361 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.361 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.361 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.362 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:10:24.362 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.727846+0000" 2024-09-22T03:10:24.362 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.362 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.362 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:10:24.362 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.677882+0000" 2024-09-22T03:10:24.362 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.362 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.362 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.362 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.362 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.362 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.363 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:10:24.363 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.363 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.363 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.363 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.363 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.363 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.363 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.363 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:10:24.363 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.678796+0000" 2024-09-22T03:10:24.363 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.363 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.364 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.364 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.364 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.364 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.364 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:10:24.364 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.364 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.364 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.364 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.364 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.364 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.364 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.365 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:10:24.365 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.680018+0000" 2024-09-22T03:10:24.365 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.365 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.365 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.365 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.365 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.365 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.365 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:10:24.365 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.365 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.365 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.366 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.366 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.366 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.366 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.366 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:10:24.366 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.366 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.366 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.366 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.366 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.366 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.366 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.367 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:10:24.367 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.367 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.367 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.367 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.367 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.367 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.367 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.367 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:10:24.367 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.682229+0000" 2024-09-22T03:10:24.367 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.368 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.368 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.368 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.368 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.368 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.368 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:10:24.368 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.368 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.368 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.368 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.368 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.685307+0000" 2024-09-22T03:10:24.369 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.369 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.369 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:10:24.369 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.702243+0000" 2024-09-22T03:10:24.369 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.369 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.369 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.369 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.369 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.369 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.369 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:10:24.369 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.370 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.370 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.370 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.370 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.370 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.370 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.370 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:10:24.370 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.726935+0000" 2024-09-22T03:10:24.370 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.370 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.370 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.370 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.371 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.371 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.371 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:10:24.371 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.371 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:10:24.371 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:10:24.371 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.371 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-22T03:10:24.371 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "client_request", 2024-09-22T03:10:24.371 INFO:teuthology.orchestra.run.smithi153.stdout: "id": 624, 2024-09-22T03:10:24.371 INFO:teuthology.orchestra.run.smithi153.stdout: "detail": { 2024-09-22T03:10:24.372 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.372 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:10:24.372 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.679767+0000" 2024-09-22T03:10:24.372 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.372 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.372 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:10:24.372 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.679777+0000" 2024-09-22T03:10:24.372 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.372 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.372 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.372 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.372 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.373 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.373 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:10:24.373 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.679909+0000" 2024-09-22T03:10:24.373 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.373 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.373 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.373 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.373 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.373 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.373 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:10:24.373 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.373 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.373 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.374 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.374 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.374 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.374 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.374 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:10:24.374 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.679994+0000" 2024-09-22T03:10:24.374 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.374 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.374 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.374 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.374 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.375 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.375 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:10:24.375 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.680710+0000" 2024-09-22T03:10:24.375 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.375 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.375 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.375 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.375 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.375 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.375 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:10:24.375 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.376 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.376 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.376 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.376 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.376 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.376 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.376 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:10:24.376 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.728756+0000" 2024-09-22T03:10:24.376 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.376 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.376 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:10:24.376 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.681928+0000" 2024-09-22T03:10:24.377 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.377 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.377 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.377 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.377 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.377 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.377 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:10:24.377 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.377 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.377 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.377 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.377 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.377 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.378 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.378 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:10:24.378 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.682800+0000" 2024-09-22T03:10:24.378 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.378 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.378 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.378 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.378 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.378 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.378 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:10:24.378 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.379 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.379 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.379 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.379 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.379 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.379 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.379 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:10:24.379 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.683104+0000" 2024-09-22T03:10:24.379 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.379 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.379 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.379 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.380 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.380 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.380 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:10:24.380 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.380 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.380 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.380 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.380 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.380 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.380 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.380 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:10:24.380 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.381 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.381 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.381 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.381 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.381 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.381 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.381 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:10:24.381 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.381 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.381 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.381 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.381 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.382 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.382 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.382 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:10:24.382 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.685738+0000" 2024-09-22T03:10:24.382 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.382 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.382 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.382 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.382 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.382 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.382 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:10:24.382 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.383 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.383 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.383 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.383 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.688566+0000" 2024-09-22T03:10:24.383 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.383 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.383 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:10:24.383 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.706966+0000" 2024-09-22T03:10:24.383 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.383 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.383 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.383 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.384 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.384 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.384 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:10:24.384 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.384 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.384 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.384 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.384 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.727200+0000" 2024-09-22T03:10:24.384 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.384 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.384 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:10:24.384 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:23.727263+0000" 2024-09-22T03:10:24.385 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.385 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.385 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.385 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.385 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.385 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.385 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:10:24.385 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.385 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:10:24.385 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:10:24.386 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.386 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-22T03:10:24.386 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "client_request", 2024-09-22T03:10:24.386 INFO:teuthology.orchestra.run.smithi153.stdout: "id": 353, 2024-09-22T03:10:24.386 INFO:teuthology.orchestra.run.smithi153.stdout: "detail": { 2024-09-22T03:10:24.386 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.386 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:10:24.386 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.604350+0000" 2024-09-22T03:10:24.386 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.386 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.386 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:10:24.386 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.604360+0000" 2024-09-22T03:10:24.387 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.387 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.387 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.387 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.387 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.387 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.387 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:10:24.387 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.605117+0000" 2024-09-22T03:10:24.387 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.387 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.387 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.387 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.388 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.388 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.388 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:10:24.388 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.388 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.388 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.388 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.388 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.388 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.388 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.388 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:10:24.388 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.605813+0000" 2024-09-22T03:10:24.389 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.389 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.389 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.389 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.389 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.389 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.389 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:10:24.389 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.606540+0000" 2024-09-22T03:10:24.389 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.389 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.389 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.389 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.390 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.390 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.390 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:10:24.390 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.390 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.390 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.390 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.390 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.390 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.390 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.390 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:10:24.390 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.648207+0000" 2024-09-22T03:10:24.391 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.391 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.391 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:10:24.391 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.607099+0000" 2024-09-22T03:10:24.391 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.391 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.391 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.391 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.391 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.391 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.391 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:10:24.391 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.392 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.392 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.392 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.392 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.392 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.392 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.392 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:10:24.392 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.607345+0000" 2024-09-22T03:10:24.392 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.392 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.392 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.392 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.393 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.393 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.393 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:10:24.393 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.393 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.393 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.393 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.393 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.393 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.393 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.393 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:10:24.394 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.607905+0000" 2024-09-22T03:10:24.394 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.394 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.394 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.394 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.394 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.394 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.394 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:10:24.394 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.394 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.394 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.394 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.395 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.395 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.395 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.395 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:10:24.395 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.395 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.395 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.395 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.395 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.395 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.395 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.395 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:10:24.396 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.396 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.396 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.396 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.396 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.396 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.396 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.396 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:10:24.396 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.608171+0000" 2024-09-22T03:10:24.396 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.396 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.396 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.396 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.397 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.397 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.397 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:10:24.397 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.397 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.397 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.397 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.397 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.608430+0000" 2024-09-22T03:10:24.397 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.397 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.397 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:10:24.397 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.610862+0000" 2024-09-22T03:10:24.398 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.398 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.398 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.398 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.398 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.398 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.398 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:10:24.398 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.398 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.398 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.398 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.398 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.399 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.399 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.399 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:10:24.399 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.647966+0000" 2024-09-22T03:10:24.399 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.399 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.399 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.399 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.399 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.399 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.399 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:10:24.399 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.400 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:10:24.400 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:10:24.400 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.400 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-22T03:10:24.400 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "client_request", 2024-09-22T03:10:24.400 INFO:teuthology.orchestra.run.smithi153.stdout: "id": 354, 2024-09-22T03:10:24.400 INFO:teuthology.orchestra.run.smithi153.stdout: "detail": { 2024-09-22T03:10:24.400 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.400 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:10:24.400 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.610711+0000" 2024-09-22T03:10:24.400 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.400 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.400 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:10:24.401 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.610721+0000" 2024-09-22T03:10:24.401 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.401 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.401 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.401 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.401 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.401 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.401 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:10:24.401 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.611021+0000" 2024-09-22T03:10:24.401 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.401 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.402 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.402 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.402 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.402 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.402 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:10:24.402 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.402 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.402 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.402 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.402 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.402 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.402 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.403 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:10:24.403 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.611167+0000" 2024-09-22T03:10:24.403 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.403 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.403 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.403 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.403 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.403 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.403 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:10:24.403 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.612613+0000" 2024-09-22T03:10:24.403 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.403 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.404 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.404 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.404 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.404 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.404 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:10:24.404 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.404 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.404 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.404 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.404 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.404 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.405 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.405 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:10:24.405 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.653340+0000" 2024-09-22T03:10:24.405 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.405 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.405 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:10:24.405 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.612853+0000" 2024-09-22T03:10:24.405 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.405 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.405 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.405 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.405 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.406 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.406 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:10:24.406 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.406 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.406 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.406 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.406 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.406 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.406 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.406 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:10:24.406 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.612975+0000" 2024-09-22T03:10:24.406 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.407 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.407 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.407 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.407 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.407 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.407 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:10:24.407 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.407 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.407 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.407 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.407 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.408 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.408 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.408 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:10:24.408 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.613074+0000" 2024-09-22T03:10:24.408 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.408 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.408 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.408 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.408 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.408 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.408 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:10:24.408 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.408 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.409 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.409 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.409 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.409 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.409 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.409 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:10:24.409 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.409 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.409 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.409 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.409 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.409 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.410 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.410 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:10:24.410 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.410 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.410 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.410 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.410 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.410 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.410 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.410 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:10:24.410 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.613226+0000" 2024-09-22T03:10:24.410 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.410 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.411 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.411 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.411 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.411 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.411 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:10:24.411 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.411 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.411 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.411 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.411 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.613535+0000" 2024-09-22T03:10:24.411 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.411 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.412 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:10:24.412 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.614825+0000" 2024-09-22T03:10:24.412 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.412 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.412 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.412 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.412 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.412 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.412 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:10:24.412 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.412 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.413 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.413 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.413 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.413 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.413 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.413 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:10:24.413 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.652964+0000" 2024-09-22T03:10:24.413 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.413 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.413 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.413 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.413 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.414 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.414 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:10:24.414 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.414 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:10:24.414 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:10:24.414 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.414 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-22T03:10:24.414 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "client_request", 2024-09-22T03:10:24.414 INFO:teuthology.orchestra.run.smithi153.stdout: "id": 356, 2024-09-22T03:10:24.414 INFO:teuthology.orchestra.run.smithi153.stdout: "detail": { 2024-09-22T03:10:24.414 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.414 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:10:24.415 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.613690+0000" 2024-09-22T03:10:24.415 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.415 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.415 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:10:24.415 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.613700+0000" 2024-09-22T03:10:24.415 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.415 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.415 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.415 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.415 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.415 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.415 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:10:24.416 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.613971+0000" 2024-09-22T03:10:24.416 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.416 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.416 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.416 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.416 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.416 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.416 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:10:24.416 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.416 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.416 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.417 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.417 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.417 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.417 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.417 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:10:24.417 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.614273+0000" 2024-09-22T03:10:24.417 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.417 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.417 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.417 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.417 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.418 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.418 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:10:24.418 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.616226+0000" 2024-09-22T03:10:24.418 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.418 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.418 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.418 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.418 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.418 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.418 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:10:24.418 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.419 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.419 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.419 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.419 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.419 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.419 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.419 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:10:24.419 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.671357+0000" 2024-09-22T03:10:24.419 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.419 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.419 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:10:24.419 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.616529+0000" 2024-09-22T03:10:24.420 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.420 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.420 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.420 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.420 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.420 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.420 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:10:24.420 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.420 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.420 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.420 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.420 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.421 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.421 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.421 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:10:24.421 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.616664+0000" 2024-09-22T03:10:24.421 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.421 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.421 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.421 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.421 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.421 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.421 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:10:24.421 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.422 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.422 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.422 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.422 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.422 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.422 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.422 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:10:24.422 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.616766+0000" 2024-09-22T03:10:24.422 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.422 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.422 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.423 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.423 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.423 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.423 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:10:24.423 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.423 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.423 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.423 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.423 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.423 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.423 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.424 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:10:24.424 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.424 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.424 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.424 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.424 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.424 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.424 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.424 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:10:24.424 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.424 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.424 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.425 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.425 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.425 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.425 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.425 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:10:24.425 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.616925+0000" 2024-09-22T03:10:24.425 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.425 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.425 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.425 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.425 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.425 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.425 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:10:24.426 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.426 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.426 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.426 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.426 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.617199+0000" 2024-09-22T03:10:24.426 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.426 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.426 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:10:24.426 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.618500+0000" 2024-09-22T03:10:24.426 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.426 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.426 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.427 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.427 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.427 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.427 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:10:24.427 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.427 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.427 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.427 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.427 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.427 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.427 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.428 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:10:24.428 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.670920+0000" 2024-09-22T03:10:24.428 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.428 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.428 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.428 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.428 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.428 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.428 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:10:24.428 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.428 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:10:24.429 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:10:24.429 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.429 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-22T03:10:24.429 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "client_request", 2024-09-22T03:10:24.429 INFO:teuthology.orchestra.run.smithi153.stdout: "id": 381, 2024-09-22T03:10:24.429 INFO:teuthology.orchestra.run.smithi153.stdout: "detail": { 2024-09-22T03:10:24.429 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.429 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:10:24.429 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.711197+0000" 2024-09-22T03:10:24.429 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.429 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.429 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:10:24.430 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.711203+0000" 2024-09-22T03:10:24.430 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.430 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.430 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.430 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.430 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.430 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.430 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:10:24.430 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.711274+0000" 2024-09-22T03:10:24.430 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.430 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.430 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.431 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.431 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.431 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.431 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:10:24.431 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.431 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.431 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.431 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.431 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.431 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.431 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.431 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:10:24.431 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.711344+0000" 2024-09-22T03:10:24.432 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.432 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.432 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.432 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.432 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.432 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.432 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:10:24.432 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.721071+0000" 2024-09-22T03:10:24.432 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.432 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.432 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.432 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.433 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.433 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.433 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:10:24.433 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.433 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.433 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.433 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.433 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.433 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.433 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.433 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:10:24.433 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.743795+0000" 2024-09-22T03:10:24.434 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.434 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.434 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:10:24.434 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.721547+0000" 2024-09-22T03:10:24.434 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.434 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.434 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.434 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.434 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.434 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.434 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:10:24.434 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.435 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.435 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.435 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.435 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.435 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.435 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.435 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:10:24.435 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.721770+0000" 2024-09-22T03:10:24.435 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.435 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.435 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.436 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.436 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.436 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.436 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:10:24.436 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.436 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.436 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.436 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.436 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.436 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.436 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.436 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:10:24.436 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.722272+0000" 2024-09-22T03:10:24.437 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.437 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.437 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.437 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.437 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.437 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.437 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:10:24.437 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.437 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.437 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.437 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.437 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.438 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.438 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.438 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:10:24.438 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.438 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.438 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.438 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.438 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.438 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.438 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.438 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:10:24.438 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.439 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.439 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.439 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.439 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.439 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.439 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.439 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:10:24.439 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.722697+0000" 2024-09-22T03:10:24.439 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.439 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.439 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.440 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.440 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.440 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.440 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:10:24.440 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.440 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.440 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.440 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.440 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.440 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.440 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.440 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:10:24.441 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.729117+0000" 2024-09-22T03:10:24.441 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.441 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.441 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.441 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.441 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.441 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.441 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:10:24.441 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.441 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.441 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.441 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.442 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.442 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.442 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.442 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:10:24.442 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.742719+0000" 2024-09-22T03:10:24.442 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.442 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.442 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.442 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.442 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.442 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.442 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:10:24.443 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.443 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:10:24.443 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:10:24.443 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.443 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-22T03:10:24.443 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "client_request", 2024-09-22T03:10:24.443 INFO:teuthology.orchestra.run.smithi153.stdout: "id": 382, 2024-09-22T03:10:24.443 INFO:teuthology.orchestra.run.smithi153.stdout: "detail": { 2024-09-22T03:10:24.443 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.443 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:10:24.443 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.711601+0000" 2024-09-22T03:10:24.443 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.444 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.444 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:10:24.444 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.711606+0000" 2024-09-22T03:10:24.444 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.444 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.444 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.444 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.444 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.444 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.444 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:10:24.444 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.711722+0000" 2024-09-22T03:10:24.444 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.444 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.445 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.445 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.445 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.445 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.445 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:10:24.445 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.445 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.445 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.445 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.445 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.445 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.445 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.446 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:10:24.446 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.711805+0000" 2024-09-22T03:10:24.446 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.446 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.446 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.446 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.446 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.446 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.446 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:10:24.446 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.721187+0000" 2024-09-22T03:10:24.446 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.446 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.447 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.447 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.447 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.447 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.447 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:10:24.447 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.447 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.447 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.447 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.447 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.447 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.447 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.448 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:10:24.448 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.758656+0000" 2024-09-22T03:10:24.448 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.448 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.448 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:10:24.448 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.722121+0000" 2024-09-22T03:10:24.448 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.448 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.448 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.448 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.448 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.448 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.448 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:10:24.449 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.449 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.449 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.449 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.449 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.449 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.449 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.449 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:10:24.449 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.722431+0000" 2024-09-22T03:10:24.449 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.449 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.449 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.450 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.450 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.450 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.450 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:10:24.450 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.450 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.450 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.450 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.450 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.450 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.450 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.450 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:10:24.451 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.723925+0000" 2024-09-22T03:10:24.451 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.451 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.451 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.451 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.451 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.451 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.451 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:10:24.451 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.451 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.451 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.451 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.451 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.452 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.452 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.452 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:10:24.452 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.452 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.452 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.452 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.452 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.452 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.452 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.452 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:10:24.452 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.453 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.453 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.453 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.453 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.453 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.453 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.453 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:10:24.453 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.724484+0000" 2024-09-22T03:10:24.453 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.453 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.453 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.453 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.454 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.454 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.454 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:10:24.454 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.454 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.454 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.454 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.454 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.454 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.454 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.454 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:10:24.454 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.743337+0000" 2024-09-22T03:10:24.455 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.455 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.455 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.455 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.455 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.455 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.455 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:10:24.455 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.455 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.455 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.455 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.455 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.455 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.456 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.456 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:10:24.456 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.758406+0000" 2024-09-22T03:10:24.456 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.456 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.456 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.456 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.456 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.456 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.456 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:10:24.456 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.456 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:10:24.457 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:10:24.457 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.457 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-22T03:10:24.457 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "client_request", 2024-09-22T03:10:24.457 INFO:teuthology.orchestra.run.smithi153.stdout: "id": 383, 2024-09-22T03:10:24.457 INFO:teuthology.orchestra.run.smithi153.stdout: "detail": { 2024-09-22T03:10:24.457 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.457 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:10:24.457 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.715614+0000" 2024-09-22T03:10:24.457 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.457 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.457 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:10:24.458 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.715621+0000" 2024-09-22T03:10:24.458 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.458 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.458 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.458 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.458 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.458 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.458 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:10:24.458 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.715727+0000" 2024-09-22T03:10:24.458 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.458 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.458 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.459 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.459 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.459 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.459 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:10:24.459 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.459 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.459 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.459 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.459 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.459 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.459 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.460 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:10:24.460 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.715782+0000" 2024-09-22T03:10:24.460 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.460 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.460 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.460 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.460 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.460 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.460 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:10:24.460 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.721605+0000" 2024-09-22T03:10:24.460 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.460 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.461 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.461 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.461 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.461 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.461 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:10:24.461 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.461 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.461 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.461 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.461 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.461 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.461 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.462 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:10:24.462 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.774419+0000" 2024-09-22T03:10:24.462 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.462 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.462 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:10:24.462 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.723061+0000" 2024-09-22T03:10:24.462 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.462 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.462 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.462 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.462 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.462 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.463 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:10:24.463 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.463 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.463 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.463 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.463 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.463 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.463 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.463 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:10:24.463 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.723853+0000" 2024-09-22T03:10:24.463 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.463 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.464 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.464 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.464 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.464 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.464 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:10:24.464 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.464 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.464 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.464 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.464 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.464 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.464 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.465 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:10:24.465 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.724590+0000" 2024-09-22T03:10:24.465 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.465 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.465 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.465 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.465 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.465 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.465 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:10:24.465 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.465 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.465 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.466 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.466 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.466 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.466 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.466 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:10:24.466 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.466 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.466 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.466 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.466 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.466 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.466 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.467 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:10:24.467 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.467 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.467 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.467 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.467 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.467 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.467 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.467 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:10:24.467 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.725264+0000" 2024-09-22T03:10:24.467 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.467 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.468 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.468 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.468 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.468 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.468 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:10:24.468 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.468 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.468 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.468 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.468 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.743427+0000" 2024-09-22T03:10:24.468 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.468 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.469 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:10:24.469 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.743502+0000" 2024-09-22T03:10:24.469 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.469 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.469 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.469 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.469 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.469 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.469 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:10:24.469 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.469 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.469 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.470 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.470 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.470 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.470 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.470 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:10:24.470 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.774178+0000" 2024-09-22T03:10:24.470 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.470 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.470 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.470 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.470 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.470 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.471 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:10:24.471 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.471 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:10:24.471 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:10:24.471 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.471 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-22T03:10:24.471 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "client_request", 2024-09-22T03:10:24.471 INFO:teuthology.orchestra.run.smithi153.stdout: "id": 384, 2024-09-22T03:10:24.471 INFO:teuthology.orchestra.run.smithi153.stdout: "detail": { 2024-09-22T03:10:24.471 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.471 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:10:24.471 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.721037+0000" 2024-09-22T03:10:24.472 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.472 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.472 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:10:24.472 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.721048+0000" 2024-09-22T03:10:24.472 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.472 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.472 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.472 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.472 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.472 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.472 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:10:24.473 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.721214+0000" 2024-09-22T03:10:24.473 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.473 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.473 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.473 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.473 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.473 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.473 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:10:24.473 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.473 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.473 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.473 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.474 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.474 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.474 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.474 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:10:24.474 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.721310+0000" 2024-09-22T03:10:24.474 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.474 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.474 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.474 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.474 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.474 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.474 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:10:24.475 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.723225+0000" 2024-09-22T03:10:24.475 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.475 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.475 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.475 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.475 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.475 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.475 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:10:24.475 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.475 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.475 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.475 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.475 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.476 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.476 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.476 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:10:24.476 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.785443+0000" 2024-09-22T03:10:24.476 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.476 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.476 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:10:24.476 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.724238+0000" 2024-09-22T03:10:24.476 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.476 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.476 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.476 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.477 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.477 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.477 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:10:24.477 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.477 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.477 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.477 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.477 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.477 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.477 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.477 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:10:24.477 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.724958+0000" 2024-09-22T03:10:24.478 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.478 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.478 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.478 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.478 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.478 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.478 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:10:24.478 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.478 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.478 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.478 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.478 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.478 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.479 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.479 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:10:24.479 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.725340+0000" 2024-09-22T03:10:24.479 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.479 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.479 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.479 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.479 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.479 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.479 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:10:24.479 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.479 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.480 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.480 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.480 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.480 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.480 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.480 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:10:24.480 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.480 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.480 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.480 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.480 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.480 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.481 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.481 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:10:24.481 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.481 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.481 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.481 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.481 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.481 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.481 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.481 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:10:24.481 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.726466+0000" 2024-09-22T03:10:24.481 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.481 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.482 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.482 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.482 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.482 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.482 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:10:24.482 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.482 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.482 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.482 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.482 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.743490+0000" 2024-09-22T03:10:24.482 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.482 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.483 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:10:24.483 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.746365+0000" 2024-09-22T03:10:24.483 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.483 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.483 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.483 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.483 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.483 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.483 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:10:24.483 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.483 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.483 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.484 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.484 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.484 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.484 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.484 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:10:24.484 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.785043+0000" 2024-09-22T03:10:24.484 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.484 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.484 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.484 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.484 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.484 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.484 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:10:24.485 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.485 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:10:24.485 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:10:24.485 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.485 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-22T03:10:24.485 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "client_request", 2024-09-22T03:10:24.485 INFO:teuthology.orchestra.run.smithi153.stdout: "id": 385, 2024-09-22T03:10:24.485 INFO:teuthology.orchestra.run.smithi153.stdout: "detail": { 2024-09-22T03:10:24.485 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.485 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:10:24.485 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.722020+0000" 2024-09-22T03:10:24.485 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.486 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.486 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:10:24.486 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.722028+0000" 2024-09-22T03:10:24.486 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.486 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.486 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.486 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.486 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.486 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.486 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:10:24.486 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.722139+0000" 2024-09-22T03:10:24.486 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.487 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.487 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.487 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.487 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.487 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.487 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:10:24.487 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.487 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.487 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.487 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.487 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.487 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.487 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.488 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:10:24.488 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.722471+0000" 2024-09-22T03:10:24.488 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.488 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.488 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.488 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.488 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.488 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.488 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:10:24.488 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.723572+0000" 2024-09-22T03:10:24.488 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.488 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.489 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.489 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.489 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.489 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.489 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:10:24.489 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.489 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.489 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.489 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.489 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.489 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.489 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.489 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:10:24.490 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.793955+0000" 2024-09-22T03:10:24.490 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.490 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.490 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:10:24.490 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.725171+0000" 2024-09-22T03:10:24.490 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.490 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.490 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.490 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.490 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.490 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.490 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:10:24.491 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.491 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.491 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.491 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.491 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.491 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.491 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.491 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:10:24.491 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.725655+0000" 2024-09-22T03:10:24.491 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.491 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.491 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.491 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.492 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.492 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.492 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:10:24.492 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.492 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.492 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.492 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.492 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.492 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.492 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.492 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:10:24.493 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.726843+0000" 2024-09-22T03:10:24.493 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.493 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.493 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.493 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.493 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.493 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.493 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:10:24.493 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.493 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.493 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.493 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.493 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.494 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.494 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.494 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:10:24.494 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.494 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.494 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.494 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.494 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.494 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.494 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.494 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:10:24.494 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.495 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.495 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.495 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.495 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.495 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.495 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.495 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:10:24.495 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.727237+0000" 2024-09-22T03:10:24.495 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.495 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.495 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.495 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.496 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.496 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.496 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:10:24.496 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.496 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.496 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.496 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.496 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.743270+0000" 2024-09-22T03:10:24.496 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.496 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.496 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:10:24.496 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.750375+0000" 2024-09-22T03:10:24.496 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.497 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.497 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.497 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.497 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.497 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.497 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:10:24.497 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.497 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.497 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.497 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.497 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.498 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.498 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.498 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:10:24.498 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.793568+0000" 2024-09-22T03:10:24.498 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.498 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.498 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.498 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.498 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.498 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.498 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:10:24.498 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.499 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:10:24.499 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:10:24.499 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.499 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-22T03:10:24.499 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "client_request", 2024-09-22T03:10:24.499 INFO:teuthology.orchestra.run.smithi153.stdout: "id": 386, 2024-09-22T03:10:24.499 INFO:teuthology.orchestra.run.smithi153.stdout: "detail": { 2024-09-22T03:10:24.499 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.499 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:10:24.499 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.722888+0000" 2024-09-22T03:10:24.499 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.499 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.500 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:10:24.500 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.722901+0000" 2024-09-22T03:10:24.500 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.500 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.501 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.501 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.501 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.501 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.501 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:10:24.501 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.722982+0000" 2024-09-22T03:10:24.501 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.501 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.501 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.501 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.501 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.501 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.502 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:10:24.502 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.502 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.502 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.502 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.502 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.502 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.502 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.502 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:10:24.502 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.723030+0000" 2024-09-22T03:10:24.502 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.502 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.502 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.503 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.503 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.503 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.503 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:10:24.503 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.723614+0000" 2024-09-22T03:10:24.503 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.503 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.503 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.503 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.503 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.503 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.503 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:10:24.504 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.504 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.504 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.504 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.504 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.504 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.504 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.504 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:10:24.504 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.795247+0000" 2024-09-22T03:10:24.504 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.504 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.504 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:10:24.505 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.726232+0000" 2024-09-22T03:10:24.505 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.505 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.505 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.505 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.505 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.505 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.505 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:10:24.505 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.505 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.505 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.505 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.505 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.506 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.506 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.506 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:10:24.506 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.726886+0000" 2024-09-22T03:10:24.506 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.506 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.506 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.506 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.506 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.506 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.506 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:10:24.507 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.507 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.507 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.507 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.507 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.507 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.507 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.507 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:10:24.507 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.727331+0000" 2024-09-22T03:10:24.507 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.507 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.507 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.508 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.508 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.508 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.508 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:10:24.508 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.508 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.508 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.508 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.508 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.508 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.508 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.508 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:10:24.509 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.509 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.509 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.509 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.509 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.509 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.509 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.509 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:10:24.509 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.509 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.509 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.509 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.510 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.510 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.510 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.510 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:10:24.510 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.727740+0000" 2024-09-22T03:10:24.510 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.510 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.510 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.510 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.510 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.510 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.510 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:10:24.511 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.511 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.511 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.511 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.511 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.743202+0000" 2024-09-22T03:10:24.511 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.511 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.511 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:10:24.511 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.752970+0000" 2024-09-22T03:10:24.511 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.511 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.511 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.512 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.512 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.512 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.512 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:10:24.512 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.512 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.512 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.512 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.512 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.512 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.512 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.512 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:10:24.513 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.794555+0000" 2024-09-22T03:10:24.513 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.513 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.513 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.513 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.513 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.513 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.513 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:10:24.513 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.513 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:10:24.513 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:10:24.513 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.514 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-22T03:10:24.514 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "client_request", 2024-09-22T03:10:24.514 INFO:teuthology.orchestra.run.smithi153.stdout: "id": 387, 2024-09-22T03:10:24.514 INFO:teuthology.orchestra.run.smithi153.stdout: "detail": { 2024-09-22T03:10:24.514 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.514 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:10:24.514 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.724326+0000" 2024-09-22T03:10:24.514 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.514 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.514 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:10:24.514 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.724333+0000" 2024-09-22T03:10:24.514 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.515 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.515 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.515 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.515 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.515 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.515 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:10:24.515 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.724456+0000" 2024-09-22T03:10:24.515 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.515 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.515 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.515 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.515 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.516 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.516 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:10:24.516 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.516 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.516 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.516 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.516 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.516 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.516 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.516 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:10:24.516 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.724550+0000" 2024-09-22T03:10:24.516 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.516 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.517 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.517 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.517 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.517 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.517 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:10:24.517 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.725430+0000" 2024-09-22T03:10:24.517 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.517 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.517 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.517 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.517 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.517 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.518 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:10:24.518 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.518 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.518 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.518 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.518 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.518 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.518 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.518 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:10:24.518 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.808237+0000" 2024-09-22T03:10:24.518 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.518 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.519 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:10:24.519 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.727090+0000" 2024-09-22T03:10:24.519 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.519 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.519 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.519 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.519 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.519 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.519 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:10:24.519 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.519 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.519 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.520 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.520 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.520 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.520 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.520 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:10:24.520 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.727902+0000" 2024-09-22T03:10:24.520 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.520 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.520 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.520 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.520 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.521 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.521 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:10:24.521 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.521 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.521 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.521 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.521 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.521 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.521 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.521 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:10:24.521 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.728423+0000" 2024-09-22T03:10:24.521 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.522 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.522 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.522 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.522 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.522 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.522 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:10:24.522 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.522 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.522 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.522 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.522 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.523 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.523 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.523 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:10:24.523 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.523 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.523 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.523 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.523 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.523 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.523 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.523 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:10:24.523 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.523 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.524 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.524 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.524 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.524 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.524 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.524 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:10:24.524 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.729537+0000" 2024-09-22T03:10:24.524 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.524 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.524 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.524 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.524 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.525 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.525 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:10:24.525 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.525 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.525 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.525 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.525 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.743086+0000" 2024-09-22T03:10:24.525 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.525 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.525 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:10:24.525 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.756343+0000" 2024-09-22T03:10:24.525 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.526 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.526 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.526 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.526 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.526 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.526 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:10:24.526 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.526 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.526 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.526 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.526 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.526 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.527 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.527 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:10:24.527 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.802342+0000" 2024-09-22T03:10:24.527 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.527 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.527 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.527 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.527 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.527 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.527 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:10:24.527 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.527 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:10:24.528 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:10:24.528 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.528 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-22T03:10:24.528 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "client_request", 2024-09-22T03:10:24.528 INFO:teuthology.orchestra.run.smithi153.stdout: "id": 388, 2024-09-22T03:10:24.528 INFO:teuthology.orchestra.run.smithi153.stdout: "detail": { 2024-09-22T03:10:24.528 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.528 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:10:24.528 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.725418+0000" 2024-09-22T03:10:24.528 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.528 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.528 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:10:24.529 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.725425+0000" 2024-09-22T03:10:24.529 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.529 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.529 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.529 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.529 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.529 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.529 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:10:24.529 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.725624+0000" 2024-09-22T03:10:24.529 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.529 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.529 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.529 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.530 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.530 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.530 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:10:24.530 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.530 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.530 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.530 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.530 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.530 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.530 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.530 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:10:24.531 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.725752+0000" 2024-09-22T03:10:24.531 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.531 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.531 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.531 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.531 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.531 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.531 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:10:24.531 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.727547+0000" 2024-09-22T03:10:24.531 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.531 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.531 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.531 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.532 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.532 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.532 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:10:24.532 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.532 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.532 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.532 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.532 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.532 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.532 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.532 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:10:24.532 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.809417+0000" 2024-09-22T03:10:24.533 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.533 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.533 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:10:24.533 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.728313+0000" 2024-09-22T03:10:24.533 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.533 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.533 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.533 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.533 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.533 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.533 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:10:24.533 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.534 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.534 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.534 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.534 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.534 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.534 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.534 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:10:24.534 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.728872+0000" 2024-09-22T03:10:24.534 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.534 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.534 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.534 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.535 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.535 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.535 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:10:24.535 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.535 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.535 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.535 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.535 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.535 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.535 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.535 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:10:24.535 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.729942+0000" 2024-09-22T03:10:24.536 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.536 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.536 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.536 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.536 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.536 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.536 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:10:24.536 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.536 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.536 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.536 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.536 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.537 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.537 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.537 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:10:24.537 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.537 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.537 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.537 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.537 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.537 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.537 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.537 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:10:24.537 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.537 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.538 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.538 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.538 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.538 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.538 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.538 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:10:24.538 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.731849+0000" 2024-09-22T03:10:24.538 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.538 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.538 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.538 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.539 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.539 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.539 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:10:24.539 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.539 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.539 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.539 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.539 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.743150+0000" 2024-09-22T03:10:24.539 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.539 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.539 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:10:24.539 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.760316+0000" 2024-09-22T03:10:24.539 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.540 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.540 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.540 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.540 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.540 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.540 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:10:24.540 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.540 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.540 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.540 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.540 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.541 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.541 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.541 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:10:24.541 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.807546+0000" 2024-09-22T03:10:24.541 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.541 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.541 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.541 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.541 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.541 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.541 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:10:24.541 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.541 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:10:24.542 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:10:24.542 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.542 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-22T03:10:24.542 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "client_request", 2024-09-22T03:10:24.542 INFO:teuthology.orchestra.run.smithi153.stdout: "id": 397, 2024-09-22T03:10:24.542 INFO:teuthology.orchestra.run.smithi153.stdout: "detail": { 2024-09-22T03:10:24.542 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.542 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:10:24.542 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.795470+0000" 2024-09-22T03:10:24.542 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.542 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.543 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:10:24.543 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.795480+0000" 2024-09-22T03:10:24.543 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.543 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.543 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.543 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.543 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.543 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.543 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:10:24.543 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.795624+0000" 2024-09-22T03:10:24.543 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.543 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.543 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.544 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.544 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.544 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.544 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:10:24.544 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.544 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.544 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.544 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.544 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.544 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.544 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.544 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:10:24.545 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.795717+0000" 2024-09-22T03:10:24.545 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.545 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.545 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.545 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.545 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.545 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.545 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:10:24.545 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.796229+0000" 2024-09-22T03:10:24.545 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.545 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.545 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.546 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.546 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.546 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.546 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:10:24.546 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.546 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.546 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.546 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.546 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.546 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.546 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.546 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:10:24.547 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.817508+0000" 2024-09-22T03:10:24.547 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.547 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.547 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:10:24.547 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.796717+0000" 2024-09-22T03:10:24.547 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.547 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.547 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.547 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.547 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.547 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.547 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:10:24.548 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.548 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.548 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.548 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.548 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.548 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.548 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.548 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:10:24.548 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.796885+0000" 2024-09-22T03:10:24.548 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.548 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.548 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.549 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.549 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.549 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.549 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:10:24.549 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.549 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.549 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.549 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.549 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.549 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.549 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.549 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:10:24.550 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.797049+0000" 2024-09-22T03:10:24.550 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.550 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.550 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.550 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.550 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.550 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.550 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:10:24.550 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.550 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.550 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.550 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.551 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.551 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.551 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.551 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:10:24.551 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.551 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.551 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.551 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.551 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.551 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.551 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.552 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:10:24.552 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.552 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.552 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.552 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.552 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.552 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.552 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.552 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:10:24.552 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.797291+0000" 2024-09-22T03:10:24.552 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.552 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.553 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.553 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.553 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.553 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.553 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:10:24.553 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.553 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.553 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.553 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.553 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.553 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.553 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.554 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:10:24.554 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.803782+0000" 2024-09-22T03:10:24.554 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.554 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.554 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.554 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.554 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.554 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.554 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:10:24.554 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.555 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.555 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.555 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.555 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.555 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.555 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.555 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:10:24.555 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.816162+0000" 2024-09-22T03:10:24.555 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.555 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.555 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.555 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.556 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.556 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.556 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:10:24.556 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.556 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:10:24.556 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:10:24.556 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.556 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-22T03:10:24.556 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "client_request", 2024-09-22T03:10:24.556 INFO:teuthology.orchestra.run.smithi153.stdout: "id": 398, 2024-09-22T03:10:24.556 INFO:teuthology.orchestra.run.smithi153.stdout: "detail": { 2024-09-22T03:10:24.556 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.557 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:10:24.557 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.796314+0000" 2024-09-22T03:10:24.557 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.557 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.557 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:10:24.557 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.796320+0000" 2024-09-22T03:10:24.557 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.557 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.557 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.557 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.557 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.557 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.558 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:10:24.558 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.796431+0000" 2024-09-22T03:10:24.558 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.558 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.558 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.558 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.558 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.558 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.558 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:10:24.558 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.558 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.559 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.559 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.559 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.559 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.559 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.559 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:10:24.559 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.796532+0000" 2024-09-22T03:10:24.559 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.559 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.559 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.559 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.560 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.561 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.561 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:10:24.562 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.797777+0000" 2024-09-22T03:10:24.562 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.562 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.562 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.562 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.562 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.562 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.562 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:10:24.562 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.562 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.562 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.563 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.563 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.563 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.563 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.563 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:10:24.563 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.844788+0000" 2024-09-22T03:10:24.563 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.563 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.563 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:10:24.563 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.798235+0000" 2024-09-22T03:10:24.563 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.563 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.564 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.564 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.564 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.564 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.564 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:10:24.564 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.564 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.564 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.564 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.564 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.564 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.565 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.565 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:10:24.565 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.798725+0000" 2024-09-22T03:10:24.565 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.565 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.565 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.565 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.565 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.565 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.565 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:10:24.565 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.565 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.566 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.566 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.566 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.566 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.566 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.566 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:10:24.566 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.798883+0000" 2024-09-22T03:10:24.566 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.566 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.566 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.566 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.567 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.567 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.567 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:10:24.567 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.567 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.567 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.567 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.567 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.567 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.567 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.567 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:10:24.567 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.568 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.568 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.568 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.568 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.568 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.568 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.568 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:10:24.568 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.568 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.568 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.568 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.568 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.569 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.569 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.569 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:10:24.569 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.799325+0000" 2024-09-22T03:10:24.569 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.569 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.569 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.569 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.569 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.569 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.569 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:10:24.569 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.570 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.570 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.570 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.570 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.570 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.570 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.570 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:10:24.570 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.817281+0000" 2024-09-22T03:10:24.570 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.570 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.570 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.570 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.571 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.571 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.571 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:10:24.571 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.571 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.571 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.571 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.571 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.571 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.571 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.571 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:10:24.571 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.844546+0000" 2024-09-22T03:10:24.572 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.572 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.572 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.572 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.572 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.572 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.572 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:10:24.572 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.572 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:10:24.572 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:10:24.572 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.573 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-22T03:10:24.573 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "client_request", 2024-09-22T03:10:24.573 INFO:teuthology.orchestra.run.smithi153.stdout: "id": 399, 2024-09-22T03:10:24.573 INFO:teuthology.orchestra.run.smithi153.stdout: "detail": { 2024-09-22T03:10:24.573 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.573 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:10:24.573 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.796886+0000" 2024-09-22T03:10:24.573 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.573 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.573 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:10:24.573 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.796897+0000" 2024-09-22T03:10:24.573 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.574 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.574 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.574 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.574 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.574 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.574 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:10:24.574 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.797019+0000" 2024-09-22T03:10:24.574 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.574 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.574 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.574 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.575 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.575 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.575 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:10:24.575 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.575 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.575 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.575 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.575 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.575 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.575 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.575 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:10:24.575 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.797136+0000" 2024-09-22T03:10:24.576 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.576 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.576 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.576 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.576 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.576 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.576 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:10:24.576 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.797869+0000" 2024-09-22T03:10:24.576 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.576 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.576 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.576 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.577 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.577 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.577 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:10:24.577 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.577 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.577 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.577 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.577 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.577 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.577 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.577 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:10:24.578 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.855354+0000" 2024-09-22T03:10:24.578 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.578 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.578 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:10:24.578 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.798646+0000" 2024-09-22T03:10:24.578 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.578 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.578 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.578 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.578 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.578 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.578 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:10:24.579 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.579 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.579 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.579 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.579 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.579 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.579 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.579 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:10:24.579 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.799481+0000" 2024-09-22T03:10:24.579 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.579 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.579 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.580 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.580 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.580 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.580 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:10:24.580 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.580 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.580 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.580 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.580 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.580 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.580 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.580 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:10:24.581 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.800690+0000" 2024-09-22T03:10:24.581 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.581 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.581 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.581 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.581 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.581 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.581 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:10:24.581 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.581 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.581 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.581 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.582 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.582 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.582 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.582 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:10:24.582 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.582 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.582 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.582 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.582 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.582 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.582 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.582 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:10:24.583 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.583 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.583 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.583 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.583 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.583 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.583 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.583 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:10:24.583 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.801147+0000" 2024-09-22T03:10:24.583 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.583 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.583 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.583 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.584 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.584 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.584 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:10:24.584 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.584 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.584 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.584 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.584 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.817157+0000" 2024-09-22T03:10:24.584 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.584 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.584 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:10:24.584 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.817437+0000" 2024-09-22T03:10:24.585 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.585 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.585 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.585 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.585 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.585 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.585 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:10:24.585 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.585 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.585 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.585 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.585 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.586 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.586 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.586 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:10:24.586 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.854993+0000" 2024-09-22T03:10:24.586 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.586 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.586 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.586 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.586 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.586 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.586 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:10:24.586 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.587 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:10:24.587 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:10:24.587 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.587 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-22T03:10:24.587 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "client_request", 2024-09-22T03:10:24.587 INFO:teuthology.orchestra.run.smithi153.stdout: "id": 400, 2024-09-22T03:10:24.587 INFO:teuthology.orchestra.run.smithi153.stdout: "detail": { 2024-09-22T03:10:24.587 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.587 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:10:24.587 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.797388+0000" 2024-09-22T03:10:24.587 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.587 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.588 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:10:24.588 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.797394+0000" 2024-09-22T03:10:24.588 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.588 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.588 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.588 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.588 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.588 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.588 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:10:24.588 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.797522+0000" 2024-09-22T03:10:24.588 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.588 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.589 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.589 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.589 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.589 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.589 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:10:24.589 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.589 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.589 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.589 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.589 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.589 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.589 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.590 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:10:24.590 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.797603+0000" 2024-09-22T03:10:24.590 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.590 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.590 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.590 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.590 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.590 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.590 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:10:24.590 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.798286+0000" 2024-09-22T03:10:24.590 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.590 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.591 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.591 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.591 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.591 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.591 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:10:24.591 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.591 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.591 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.591 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.591 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.591 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.591 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.591 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:10:24.592 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.858674+0000" 2024-09-22T03:10:24.592 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.592 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.592 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:10:24.592 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.800010+0000" 2024-09-22T03:10:24.592 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.592 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.592 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.592 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.592 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.592 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.592 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:10:24.593 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.593 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.593 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.593 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.593 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.593 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.593 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.593 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:10:24.593 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.800579+0000" 2024-09-22T03:10:24.593 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.593 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.593 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.593 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.594 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.594 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.594 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:10:24.594 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.594 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.594 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.594 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.594 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.594 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.594 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.594 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:10:24.594 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.801233+0000" 2024-09-22T03:10:24.595 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.595 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.595 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.595 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.595 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.595 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.595 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:10:24.595 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.595 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.595 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.595 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.595 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.596 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.596 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.596 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:10:24.596 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.596 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.596 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.596 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.596 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.596 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.596 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.596 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:10:24.596 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.597 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.597 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.597 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.597 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.597 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.597 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.597 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:10:24.597 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.802436+0000" 2024-09-22T03:10:24.597 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.597 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.597 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.597 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.598 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.598 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.598 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:10:24.598 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.598 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.598 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.598 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.598 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.817060+0000" 2024-09-22T03:10:24.598 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.598 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.598 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:10:24.598 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.827571+0000" 2024-09-22T03:10:24.599 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.599 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.599 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.599 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.599 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.599 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.599 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:10:24.599 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.599 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.599 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.599 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.600 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.600 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.600 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.600 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:10:24.600 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.858432+0000" 2024-09-22T03:10:24.600 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.600 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.600 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.600 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.600 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.600 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.600 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:10:24.601 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.601 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:10:24.601 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:10:24.601 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.601 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-22T03:10:24.601 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "client_request", 2024-09-22T03:10:24.601 INFO:teuthology.orchestra.run.smithi153.stdout: "id": 401, 2024-09-22T03:10:24.601 INFO:teuthology.orchestra.run.smithi153.stdout: "detail": { 2024-09-22T03:10:24.601 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.601 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:10:24.601 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.799447+0000" 2024-09-22T03:10:24.601 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.601 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.602 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:10:24.602 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.799456+0000" 2024-09-22T03:10:24.602 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.602 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.602 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.602 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.602 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.602 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.602 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:10:24.602 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.799595+0000" 2024-09-22T03:10:24.602 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.603 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.603 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.603 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.603 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.603 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.603 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:10:24.603 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.603 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.603 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.603 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.603 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.604 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.604 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.604 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:10:24.604 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.799684+0000" 2024-09-22T03:10:24.604 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.604 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.604 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.604 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.604 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.604 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.604 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:10:24.604 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.800958+0000" 2024-09-22T03:10:24.604 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.605 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.605 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.605 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.605 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.605 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.605 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:10:24.605 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.605 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.605 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.605 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.605 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.605 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.606 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.606 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:10:24.606 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.859394+0000" 2024-09-22T03:10:24.606 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.606 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.606 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:10:24.606 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.801737+0000" 2024-09-22T03:10:24.606 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.606 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.606 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.606 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.606 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.607 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.607 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:10:24.607 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.607 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.607 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.607 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.607 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.607 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.607 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.607 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:10:24.607 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.802131+0000" 2024-09-22T03:10:24.607 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.607 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.608 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.608 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.608 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.608 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.608 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:10:24.608 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.608 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.608 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.608 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.608 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.608 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.608 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.609 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:10:24.609 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.803766+0000" 2024-09-22T03:10:24.609 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.609 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.609 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.609 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.609 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.609 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.609 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:10:24.609 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.609 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.609 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.610 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.610 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.610 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.610 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.610 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:10:24.610 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.610 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.610 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.610 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.610 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.610 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.610 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.610 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:10:24.611 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.611 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.611 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.611 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.611 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.611 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.611 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.611 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:10:24.611 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.807865+0000" 2024-09-22T03:10:24.611 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.611 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.611 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.612 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.612 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.612 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.612 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:10:24.612 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.612 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.612 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.612 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.612 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.816662+0000" 2024-09-22T03:10:24.612 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.612 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.612 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:10:24.612 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.829869+0000" 2024-09-22T03:10:24.613 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.613 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.613 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.613 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.613 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.613 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.613 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:10:24.613 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.613 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.613 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.613 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.613 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.614 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.614 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.614 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:10:24.614 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.859158+0000" 2024-09-22T03:10:24.614 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.614 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.614 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.614 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.614 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.614 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.614 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:10:24.614 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.615 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:10:24.615 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:10:24.615 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.615 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-22T03:10:24.615 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "client_request", 2024-09-22T03:10:24.615 INFO:teuthology.orchestra.run.smithi153.stdout: "id": 402, 2024-09-22T03:10:24.615 INFO:teuthology.orchestra.run.smithi153.stdout: "detail": { 2024-09-22T03:10:24.615 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.615 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:10:24.615 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.800197+0000" 2024-09-22T03:10:24.615 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.615 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.616 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:10:24.616 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.800204+0000" 2024-09-22T03:10:24.616 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.616 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.616 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.616 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.616 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.616 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.616 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:10:24.616 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.800329+0000" 2024-09-22T03:10:24.616 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.616 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.616 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.617 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.617 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.617 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.617 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:10:24.617 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.617 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.617 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.617 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.617 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.617 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.617 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.617 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:10:24.618 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.800438+0000" 2024-09-22T03:10:24.618 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.618 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.618 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.618 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.618 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.618 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.618 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:10:24.618 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.801007+0000" 2024-09-22T03:10:24.618 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.618 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.618 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.619 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.619 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.619 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.619 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:10:24.619 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.619 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.619 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.619 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.619 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.619 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.619 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.619 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:10:24.620 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.863616+0000" 2024-09-22T03:10:24.620 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.620 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.620 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:10:24.620 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.803486+0000" 2024-09-22T03:10:24.620 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.620 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.620 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.620 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.620 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.620 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.620 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:10:24.620 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.621 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.621 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.621 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.621 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.621 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.621 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.621 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:10:24.621 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.804192+0000" 2024-09-22T03:10:24.621 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.621 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.621 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.621 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.622 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.622 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.622 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:10:24.622 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.622 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.622 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.622 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.622 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.622 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.622 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.622 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:10:24.622 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.807514+0000" 2024-09-22T03:10:24.623 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.623 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.623 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.623 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.623 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.623 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.623 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:10:24.623 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.623 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.623 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.623 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.623 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.624 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.624 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.624 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:10:24.624 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.624 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.624 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.624 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.624 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.624 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.624 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.624 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:10:24.624 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.624 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.625 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.625 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.625 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.625 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.625 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.625 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:10:24.625 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.809255+0000" 2024-09-22T03:10:24.625 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.625 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.625 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.625 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.625 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.625 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.626 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:10:24.626 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.626 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.626 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.626 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.626 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.816770+0000" 2024-09-22T03:10:24.626 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.626 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.626 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:10:24.626 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.831816+0000" 2024-09-22T03:10:24.626 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.627 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.627 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.627 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.627 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.627 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.627 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:10:24.627 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.627 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.627 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.627 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.627 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.627 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.627 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.628 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:10:24.628 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.862448+0000" 2024-09-22T03:10:24.628 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.628 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.628 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.628 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.628 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.628 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.628 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:10:24.628 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.628 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:10:24.628 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:10:24.629 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.629 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-22T03:10:24.629 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "client_request", 2024-09-22T03:10:24.629 INFO:teuthology.orchestra.run.smithi153.stdout: "id": 403, 2024-09-22T03:10:24.629 INFO:teuthology.orchestra.run.smithi153.stdout: "detail": { 2024-09-22T03:10:24.629 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.629 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:10:24.629 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.802559+0000" 2024-09-22T03:10:24.629 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.629 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.629 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:10:24.629 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.802568+0000" 2024-09-22T03:10:24.629 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.630 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.630 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.630 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.630 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.630 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.630 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:10:24.630 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.802706+0000" 2024-09-22T03:10:24.630 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.630 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.630 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.630 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.630 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.631 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.631 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:10:24.631 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.631 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.631 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.631 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.631 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.631 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.631 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.631 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:10:24.631 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.802801+0000" 2024-09-22T03:10:24.632 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.632 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.632 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.632 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.632 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.632 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.632 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:10:24.632 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.803652+0000" 2024-09-22T03:10:24.632 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.632 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.632 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.632 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.632 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.633 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.633 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:10:24.633 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.633 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.633 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.633 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.633 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.633 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.633 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.633 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:10:24.633 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.870990+0000" 2024-09-22T03:10:24.633 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.634 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.634 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:10:24.634 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.805322+0000" 2024-09-22T03:10:24.634 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.634 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.634 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.634 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.634 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.634 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.634 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:10:24.634 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.634 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.634 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.635 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.635 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.635 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.635 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.635 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:10:24.635 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.808158+0000" 2024-09-22T03:10:24.635 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.635 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.635 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.635 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.635 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.635 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.636 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:10:24.636 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.636 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.636 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.636 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.636 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.636 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.636 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.636 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:10:24.636 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.810131+0000" 2024-09-22T03:10:24.636 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.636 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.636 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.637 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.637 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.637 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.637 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:10:24.637 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.637 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.637 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.637 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.637 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.637 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.637 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.637 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:10:24.638 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.638 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.638 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.638 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.638 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.638 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.638 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.638 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:10:24.638 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.638 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.638 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.638 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.639 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.639 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.639 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.639 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:10:24.639 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.811119+0000" 2024-09-22T03:10:24.639 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.639 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.639 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.639 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.639 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.639 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.639 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:10:24.639 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.640 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.640 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.640 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.640 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.816963+0000" 2024-09-22T03:10:24.640 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.640 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.640 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:10:24.640 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.835314+0000" 2024-09-22T03:10:24.640 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.640 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.640 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.640 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.641 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.641 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.641 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:10:24.641 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.641 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.641 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.641 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.641 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.641 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.641 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.641 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:10:24.641 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.869797+0000" 2024-09-22T03:10:24.642 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.642 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.642 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.642 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.642 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.642 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.642 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:10:24.642 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.642 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:10:24.642 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:10:24.642 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.642 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-22T03:10:24.642 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "client_request", 2024-09-22T03:10:24.643 INFO:teuthology.orchestra.run.smithi153.stdout: "id": 404, 2024-09-22T03:10:24.643 INFO:teuthology.orchestra.run.smithi153.stdout: "detail": { 2024-09-22T03:10:24.643 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.643 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:10:24.643 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.808299+0000" 2024-09-22T03:10:24.643 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.643 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.643 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:10:24.643 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.808310+0000" 2024-09-22T03:10:24.643 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.643 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.643 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.644 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.644 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.644 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.644 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:10:24.644 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.808449+0000" 2024-09-22T03:10:24.644 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.644 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.644 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.644 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.644 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.644 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.644 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:10:24.645 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.645 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.645 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.645 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.645 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.645 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.645 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.645 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:10:24.645 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.808535+0000" 2024-09-22T03:10:24.645 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.645 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.645 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.645 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.646 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.646 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.646 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:10:24.646 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.810018+0000" 2024-09-22T03:10:24.646 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.646 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.646 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.646 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.646 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.646 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.646 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:10:24.646 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.647 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.647 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.647 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.647 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.647 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.647 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.647 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:10:24.647 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.876911+0000" 2024-09-22T03:10:24.647 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.647 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.647 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:10:24.647 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.810941+0000" 2024-09-22T03:10:24.648 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.648 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.648 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.648 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.648 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.648 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.648 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:10:24.648 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.648 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.648 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.648 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.648 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.648 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.649 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.649 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:10:24.649 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.811292+0000" 2024-09-22T03:10:24.649 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.649 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.649 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.649 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.649 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.649 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.649 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:10:24.649 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.649 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.649 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.650 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.650 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.650 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.650 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.650 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:10:24.650 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.811931+0000" 2024-09-22T03:10:24.650 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.650 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.650 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.650 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.650 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.650 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.651 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:10:24.651 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.651 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.651 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.651 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.651 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.651 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.651 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.651 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:10:24.651 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.651 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.651 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.652 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.652 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.652 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.652 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.652 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:10:24.652 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.652 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.652 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.652 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.652 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.652 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.652 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.652 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:10:24.653 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.812163+0000" 2024-09-22T03:10:24.653 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.653 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.653 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.653 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.653 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.653 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.653 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:10:24.653 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.653 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.653 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.653 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.654 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.817266+0000" 2024-09-22T03:10:24.654 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.654 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.654 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:10:24.654 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.838860+0000" 2024-09-22T03:10:24.654 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.654 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.654 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.654 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.654 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.654 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.654 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:10:24.655 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.655 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.655 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.655 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.655 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.655 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.655 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:10:24.655 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:10:24.655 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:10:22.876620+0000" 2024-09-22T03:10:24.655 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.655 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.655 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:10:24.656 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.656 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:10:24.656 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:10:24.656 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:10:24.656 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:10:24.656 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:10:24.656 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:10:24.656 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:10:24.656 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-22T03:10:24.656 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-22T03:10:24.657 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-22T03:10:24.658 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 26 16 4478 4462 1.34061 2.25781 0.0224529 0.0931781 2024-09-22T03:10:24.827 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:10:24.823+0000 7f147ef97640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:10:24.831 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:10:24.830+0000 7f147ef97640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:10:25.298 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 27 16 4829 4813 1.39251 2.74219 0.0363138 0.0896866 2024-09-22T03:10:26.298 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 28 16 5199 5183 1.446 2.89062 0.0459642 0.0863624 2024-09-22T03:10:26.976 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-22T03:10:27.100 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:10:27.096+0000 7f6b21903640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:10:27.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:10:27.103+0000 7f6b21903640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:10:27.298 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 29 16 5539 5523 1.48773 2.65625 0.0545326 0.0839431 2024-09-22T03:10:28.298 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 30 16 5900 5884 1.53214 2.82031 0.0524763 0.0815121 2024-09-22T03:10:29.291 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-22T03:10:29.298 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 31 16 6266 6250 1.57494 2.85938 0.0449684 0.0793295 2024-09-22T03:10:29.423 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:10:29.419+0000 7fb3d94d7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:10:29.427 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:10:29.425+0000 7fb3d94d7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:10:29.659 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [2, 3, 0, 1] 2024-09-22T03:10:29.727 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-22T03:10:29.727 INFO:tasks.thrashosds.thrasher:Killing osd 1, live_osds are [2, 3, 0, 1] 2024-09-22T03:10:29.727 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-09-22T03:10:29.727 INFO:teuthology.orchestra.run:waiting for 300 2024-09-22T03:10:30.298 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 32 16 6426 6410 1.56478 1.25 0.0177615 0.078414 2024-09-22T03:10:31.217 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~0s 2024-09-22T03:10:31.299 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 33 16 6426 6410 1.51736 0 - 0.078414 2024-09-22T03:10:31.636 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-22T03:10:31.751 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:10:31.748+0000 7fd9328d7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:10:31.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:10:31.752+0000 7fd9328d7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:10:32.299 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 34 16 6426 6410 1.47273 0 - 0.078414 2024-09-22T03:10:33.299 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 35 16 6426 6410 1.43065 0 - 0.078414 2024-09-22T03:10:34.299 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 36 16 6426 6410 1.39091 0 - 0.078414 2024-09-22T03:10:35.299 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 37 16 6426 6410 1.35332 0 - 0.078414 2024-09-22T03:10:35.760 INFO:tasks.ceph.osd.1:Stopped 2024-09-22T03:10:36.299 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 38 16 6426 6410 1.3177 0 - 0.078414 2024-09-22T03:10:36.721 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 has been restored 2024-09-22T03:10:37.299 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 39 16 6426 6410 1.28392 0 - 0.078414 2024-09-22T03:10:38.299 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:2024-09-22T03:10:38.299061+0000 min lat: 0.00849108 max lat: 13.8745 avg lat: 0.078414 2024-09-22T03:10:38.300 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-22T03:10:38.300 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 40 16 6426 6410 1.25182 0 - 0.078414 2024-09-22T03:10:39.300 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 41 16 6426 6410 1.22128 0 - 0.078414 2024-09-22T03:10:40.300 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 42 16 6426 6410 1.19221 0 - 0.078414 2024-09-22T03:10:40.760 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [1] live_osds: [2, 3, 0] 2024-09-22T03:10:40.761 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-22T03:10:40.761 INFO:tasks.thrashosds.thrasher:Reviving osd 1 2024-09-22T03:10:40.761 INFO:tasks.ceph.osd.1:Restarting daemon 2024-09-22T03:10:40.761 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2>> /var/log/ceph/ceph-osd.1.log 2024-09-22T03:10:40.764 INFO:tasks.ceph.osd.1:Started 2024-09-22T03:10:40.764 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-22T03:10:40.875 INFO:teuthology.orchestra.run.smithi153.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-09-22T03:10:40.878 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-22T03:10:40.879 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2024-09-22T03:10:41.300 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 43 16 6426 6410 1.16448 0 - 0.078414 2024-09-22T03:10:42.300 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 44 16 6426 6410 1.13801 0 - 0.078414 2024-09-22T03:10:43.300 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 45 16 6426 6410 1.11272 0 - 0.078414 2024-09-22T03:10:44.300 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 46 16 6426 6410 1.08854 0 - 0.078414 2024-09-22T03:10:45.300 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 47 16 6426 6410 1.06537 0 - 0.078414 2024-09-22T03:10:45.880 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-22T03:10:45.980 INFO:teuthology.orchestra.run.smithi153.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-09-22T03:10:45.983 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-22T03:10:45.983 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2024-09-22T03:10:46.300 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 48 16 6426 6410 1.04318 0 - 0.078414 2024-09-22T03:10:47.301 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 49 16 6426 6410 1.02189 0 - 0.078414 2024-09-22T03:10:48.301 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 50 16 6426 6410 1.00145 0 - 0.078414 2024-09-22T03:10:49.301 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 51 16 6426 6410 0.981816 0 - 0.078414 2024-09-22T03:10:50.301 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 52 16 6426 6410 0.962935 0 - 0.078414 2024-09-22T03:10:50.880 INFO:teuthology.orchestra.run.smithi153.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-09-22T03:10:50.969 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-22T03:10:50.969 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-22T03:10:50.984 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-22T03:10:51.085 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-22T03:10:51.085 INFO:teuthology.orchestra.run.smithi153.stdout: "ops_in_flight": [] 2024-09-22T03:10:51.085 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-22T03:10:51.301 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 53 16 6426 6410 0.944766 0 - 0.078414 2024-09-22T03:10:52.301 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 54 16 6426 6410 0.927271 0 - 0.078414 2024-09-22T03:10:52.970 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-22T03:10:53.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:10:53.108+0000 7febd0357640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:10:53.116 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:10:53.115+0000 7febd0357640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:10:53.301 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 55 16 6503 6487 0.921348 0.0261549 0.0501775 0.135597 2024-09-22T03:10:54.301 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 56 16 6861 6845 0.954834 2.79688 0.0460725 0.130836 2024-09-22T03:10:55.301 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 57 16 7183 7167 0.982213 2.51562 0.0392552 0.127172 2024-09-22T03:10:55.361 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-22T03:10:55.504 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:10:55.500+0000 7f8fdeb17640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:10:55.507 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:10:55.506+0000 7f8fdeb17640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:10:56.095 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [2, 3, 0, 1] 2024-09-22T03:10:56.095 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-22T03:10:56.096 INFO:tasks.thrashosds.thrasher:Killing osd 3, live_osds are [2, 3, 0, 1] 2024-09-22T03:10:56.096 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-09-22T03:10:56.096 INFO:teuthology.orchestra.run:waiting for 300 2024-09-22T03:10:56.301 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 58 16 7457 7441 1.00218 2.14062 0.0501541 0.124347 2024-09-22T03:10:57.302 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 59 16 7457 7441 0.985194 0 - 0.124347 2024-09-22T03:10:57.690 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-22T03:10:57.805 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:10:57.802+0000 7fc9fc62c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:10:57.809 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:10:57.807+0000 7fc9fc62c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:10:58.302 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:2024-09-22T03:10:58.301339+0000 min lat: 0.00849108 max lat: 23.3792 avg lat: 0.124347 2024-09-22T03:10:58.302 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-22T03:10:58.302 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 60 16 7457 7441 0.968774 0 - 0.124347 2024-09-22T03:10:59.039 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~0s 2024-09-22T03:10:59.302 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 61 16 7457 7441 0.952891 0 - 0.124347 2024-09-22T03:11:00.303 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 62 16 7457 7441 0.937521 0 - 0.124347 2024-09-22T03:11:01.303 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 63 16 7457 7441 0.92264 0 - 0.124347 2024-09-22T03:11:02.197 INFO:tasks.ceph.osd.3:Stopped 2024-09-22T03:11:02.302 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 64 16 7457 7441 0.908223 0 - 0.124347 2024-09-22T03:11:03.303 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 65 16 7457 7441 0.89425 0 - 0.124347 2024-09-22T03:11:04.303 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 66 16 7457 7441 0.8807 0 - 0.124347 2024-09-22T03:11:04.545 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 has been restored 2024-09-22T03:11:05.303 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 67 16 7457 7441 0.867556 0 - 0.124347 2024-09-22T03:11:06.303 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 68 16 7457 7441 0.854797 0 - 0.124347 2024-09-22T03:11:07.198 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [3] live_osds: [2, 0, 1] 2024-09-22T03:11:07.198 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-22T03:11:07.198 INFO:tasks.thrashosds.thrasher:Reviving osd 3 2024-09-22T03:11:07.199 INFO:tasks.ceph.osd.3:Restarting daemon 2024-09-22T03:11:07.199 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2>> /var/log/ceph/ceph-osd.3.log 2024-09-22T03:11:07.202 INFO:tasks.ceph.osd.3:Started 2024-09-22T03:11:07.202 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-22T03:11:07.303 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 69 16 7457 7441 0.842409 0 - 0.124347 2024-09-22T03:11:07.312 INFO:teuthology.orchestra.run.smithi196.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-09-22T03:11:07.315 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-22T03:11:07.315 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-09-22T03:11:08.303 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 70 16 7457 7441 0.830374 0 - 0.124347 2024-09-22T03:11:09.303 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 71 16 7457 7441 0.818678 0 - 0.124347 2024-09-22T03:11:10.303 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 72 16 7457 7441 0.807307 0 - 0.124347 2024-09-22T03:11:11.304 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 73 16 7457 7441 0.796247 0 - 0.124347 2024-09-22T03:11:12.304 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 74 16 7457 7441 0.785487 0 - 0.124347 2024-09-22T03:11:12.316 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-22T03:11:12.412 INFO:teuthology.orchestra.run.smithi196.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-09-22T03:11:12.415 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-22T03:11:12.415 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-09-22T03:11:13.304 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 75 16 7457 7441 0.775014 0 - 0.124347 2024-09-22T03:11:13.876 INFO:teuthology.orchestra.run.smithi153.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-09-22T03:11:13.881 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-22T03:11:13.881 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-22T03:11:14.304 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 76 16 7457 7441 0.764816 0 - 0.124347 2024-09-22T03:11:15.304 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 77 16 7457 7441 0.754883 0 - 0.124347 2024-09-22T03:11:15.882 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-22T03:11:16.031 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:11:16.028+0000 7fa7224d7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:11:16.038 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:11:16.037+0000 7fa7224d7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:11:16.304 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 78 16 7564 7548 0.755921 0.0417969 0.0330846 0.165313 2024-09-22T03:11:17.304 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 79 16 7914 7898 0.780962 2.73438 0.0192776 0.160011 2024-09-22T03:11:17.416 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-22T03:11:17.578 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-09-22T03:11:17.578 INFO:teuthology.orchestra.run.smithi196.stdout: "ops_in_flight": [ 2024-09-22T03:11:17.578 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-22T03:11:17.578 INFO:teuthology.orchestra.run.smithi196.stdout: "type": "replicated_request", 2024-09-22T03:11:17.578 INFO:teuthology.orchestra.run.smithi196.stdout: "id": 618, 2024-09-22T03:11:17.578 INFO:teuthology.orchestra.run.smithi196.stdout: "detail": { 2024-09-22T03:11:17.578 INFO:teuthology.orchestra.run.smithi196.stdout: "RepRequest": { 2024-09-22T03:11:17.578 INFO:teuthology.orchestra.run.smithi196.stdout: "reqid": "client.4357.0:7999", 2024-09-22T03:11:17.578 INFO:teuthology.orchestra.run.smithi196.stdout: "pgid": "4.e", 2024-09-22T03:11:17.579 INFO:teuthology.orchestra.run.smithi196.stdout: "map_epoch": 44, 2024-09-22T03:11:17.579 INFO:teuthology.orchestra.run.smithi196.stdout: "min_epoch": 43, 2024-09-22T03:11:17.579 INFO:teuthology.orchestra.run.smithi196.stdout: "oid": "4:716705c6:::benchmark_data_smithi153_32078_object999:head", 2024-09-22T03:11:17.579 INFO:teuthology.orchestra.run.smithi196.stdout: "from": "2" 2024-09-22T03:11:17.579 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:11:17.579 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:11:17.579 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.579 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-22T03:11:17.579 INFO:teuthology.orchestra.run.smithi196.stdout: "type": "replicated_request", 2024-09-22T03:11:17.579 INFO:teuthology.orchestra.run.smithi196.stdout: "id": 619, 2024-09-22T03:11:17.580 INFO:teuthology.orchestra.run.smithi196.stdout: "detail": { 2024-09-22T03:11:17.580 INFO:teuthology.orchestra.run.smithi196.stdout: "RepRequest": { 2024-09-22T03:11:17.580 INFO:teuthology.orchestra.run.smithi196.stdout: "reqid": "client.4357.0:8000", 2024-09-22T03:11:17.580 INFO:teuthology.orchestra.run.smithi196.stdout: "pgid": "4.e", 2024-09-22T03:11:17.580 INFO:teuthology.orchestra.run.smithi196.stdout: "map_epoch": 44, 2024-09-22T03:11:17.580 INFO:teuthology.orchestra.run.smithi196.stdout: "min_epoch": 43, 2024-09-22T03:11:17.580 INFO:teuthology.orchestra.run.smithi196.stdout: "oid": "4:716705c6:::benchmark_data_smithi153_32078_object999:head", 2024-09-22T03:11:17.580 INFO:teuthology.orchestra.run.smithi196.stdout: "from": "2" 2024-09-22T03:11:17.580 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:11:17.580 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:11:17.580 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.580 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-22T03:11:17.580 INFO:teuthology.orchestra.run.smithi196.stdout: "type": "client_request", 2024-09-22T03:11:17.580 INFO:teuthology.orchestra.run.smithi196.stdout: "id": 433, 2024-09-22T03:11:17.580 INFO:teuthology.orchestra.run.smithi196.stdout: "detail": { 2024-09-22T03:11:17.581 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.581 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:11:17.581 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:16.891857+0000" 2024-09-22T03:11:17.581 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.581 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.581 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:11:17.581 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:16.891860+0000" 2024-09-22T03:11:17.581 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.581 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.581 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.581 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.581 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.581 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.581 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:11:17.582 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:16.891918+0000" 2024-09-22T03:11:17.582 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.582 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.582 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.582 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.582 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.582 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.582 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:11:17.582 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.583 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.583 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.583 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.583 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.583 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.583 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.583 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:11:17.583 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:16.891965+0000" 2024-09-22T03:11:17.583 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.583 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.583 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.583 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.583 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.583 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.584 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:11:17.584 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:16.894222+0000" 2024-09-22T03:11:17.584 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.584 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.584 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.584 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.584 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.584 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.584 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:11:17.584 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.584 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.584 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.584 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.584 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.585 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.585 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.585 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:11:17.585 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:16.937677+0000" 2024-09-22T03:11:17.585 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.585 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.585 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:11:17.586 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:16.895858+0000" 2024-09-22T03:11:17.586 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.586 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.586 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.586 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.586 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.586 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.586 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:11:17.587 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.587 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.587 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.587 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.587 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.587 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.587 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.587 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:11:17.587 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:16.896388+0000" 2024-09-22T03:11:17.587 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.587 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.588 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.588 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.588 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.588 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.588 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:11:17.588 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.588 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.588 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.588 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.588 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.588 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.589 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.589 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:11:17.589 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:16.896904+0000" 2024-09-22T03:11:17.589 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.589 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.589 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.589 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.589 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.589 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.589 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:11:17.589 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.589 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.589 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.589 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.589 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.590 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.590 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.590 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:11:17.590 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.590 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.590 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.590 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.590 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.590 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.590 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.590 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:11:17.590 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.590 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.590 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.591 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.591 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.591 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.591 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.591 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:11:17.591 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:16.897648+0000" 2024-09-22T03:11:17.591 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.591 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.591 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.591 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.591 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.591 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.592 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:11:17.592 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.592 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.592 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.592 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.592 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:16.902515+0000" 2024-09-22T03:11:17.592 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.592 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.592 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:11:17.592 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:16.914104+0000" 2024-09-22T03:11:17.592 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.593 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.593 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.593 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.593 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.593 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.593 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:11:17.593 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.593 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.593 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.593 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.593 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.593 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.593 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.593 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:11:17.594 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:16.937278+0000" 2024-09-22T03:11:17.594 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.594 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.594 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.594 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.594 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.594 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.594 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:11:17.594 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.594 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:11:17.594 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:11:17.594 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.594 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-22T03:11:17.595 INFO:teuthology.orchestra.run.smithi196.stdout: "type": "client_request", 2024-09-22T03:11:17.595 INFO:teuthology.orchestra.run.smithi196.stdout: "id": 434, 2024-09-22T03:11:17.595 INFO:teuthology.orchestra.run.smithi196.stdout: "detail": { 2024-09-22T03:11:17.595 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.595 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:11:17.595 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:16.892512+0000" 2024-09-22T03:11:17.595 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.595 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.595 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:11:17.595 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:16.892515+0000" 2024-09-22T03:11:17.595 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.595 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.595 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.595 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.595 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.596 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.596 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:11:17.596 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:16.892596+0000" 2024-09-22T03:11:17.596 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.596 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.596 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.596 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.596 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.596 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.596 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:11:17.596 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.596 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.596 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.596 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.597 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.597 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.597 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.597 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:11:17.597 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:16.892649+0000" 2024-09-22T03:11:17.597 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.597 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.597 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.597 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.597 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.597 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.597 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:11:17.597 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:16.894339+0000" 2024-09-22T03:11:17.597 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.598 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.598 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.598 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.598 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.598 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.598 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:11:17.598 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.598 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.598 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.598 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.598 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.598 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.598 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.598 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:11:17.598 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:16.946331+0000" 2024-09-22T03:11:17.599 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.599 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.599 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:11:17.599 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:16.896824+0000" 2024-09-22T03:11:17.599 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.599 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.599 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.599 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.599 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.599 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.599 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:11:17.599 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.599 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.599 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.600 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.600 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.600 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.600 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.600 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:11:17.600 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:16.897130+0000" 2024-09-22T03:11:17.600 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.600 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.600 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.600 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.600 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.600 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.600 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:11:17.600 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.600 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.601 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.601 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.601 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.601 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.601 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.601 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:11:17.601 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:16.897813+0000" 2024-09-22T03:11:17.601 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.601 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.601 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.601 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.601 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.601 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.601 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:11:17.601 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.602 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.602 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.602 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.602 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.602 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.602 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.602 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:11:17.602 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.602 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.602 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.602 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.602 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.602 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.602 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.602 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:11:17.603 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.603 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.603 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.603 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.603 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.603 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.603 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.603 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:11:17.603 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:16.898550+0000" 2024-09-22T03:11:17.603 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.603 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.603 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.603 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.603 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.604 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.604 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:11:17.604 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.604 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.604 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.604 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.604 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:16.902670+0000" 2024-09-22T03:11:17.604 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.604 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.604 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:11:17.604 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:16.916023+0000" 2024-09-22T03:11:17.604 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.604 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.604 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.605 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.605 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.605 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.605 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:11:17.605 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.605 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.605 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.605 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.605 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.605 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.605 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.605 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:11:17.605 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:16.946091+0000" 2024-09-22T03:11:17.605 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.605 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.606 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.606 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.606 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.606 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.606 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:11:17.606 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.606 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:11:17.606 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:11:17.606 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.606 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-22T03:11:17.607 INFO:teuthology.orchestra.run.smithi196.stdout: "type": "client_request", 2024-09-22T03:11:17.607 INFO:teuthology.orchestra.run.smithi196.stdout: "id": 435, 2024-09-22T03:11:17.607 INFO:teuthology.orchestra.run.smithi196.stdout: "detail": { 2024-09-22T03:11:17.607 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.607 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:11:17.607 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:16.893035+0000" 2024-09-22T03:11:17.607 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.607 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.607 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:11:17.607 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:16.893038+0000" 2024-09-22T03:11:17.607 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.607 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.607 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.607 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.608 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.608 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.608 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:11:17.608 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:16.893177+0000" 2024-09-22T03:11:17.608 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.608 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.608 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.608 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.608 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.608 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.608 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:11:17.608 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.608 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.608 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.609 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.609 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.609 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.609 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.609 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:11:17.609 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:16.893254+0000" 2024-09-22T03:11:17.609 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.609 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.609 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.609 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.610 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.610 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.610 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:11:17.610 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:16.894443+0000" 2024-09-22T03:11:17.610 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.610 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.610 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.610 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.610 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.610 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.610 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:11:17.610 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.610 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.610 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.611 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.611 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.611 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.611 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.611 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:11:17.611 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:16.947043+0000" 2024-09-22T03:11:17.611 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.611 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.611 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:11:17.611 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:16.897506+0000" 2024-09-22T03:11:17.611 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.611 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.611 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.611 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.612 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.612 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.612 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:11:17.612 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.612 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.612 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.612 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.612 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.612 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.612 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.612 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:11:17.612 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:16.897987+0000" 2024-09-22T03:11:17.612 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.612 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.612 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.613 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.613 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.613 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.613 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:11:17.613 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.613 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.613 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.613 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.613 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.613 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.613 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.613 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:11:17.613 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:16.898643+0000" 2024-09-22T03:11:17.613 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.613 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.614 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.614 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.614 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.614 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.614 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:11:17.614 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.614 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.614 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.614 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.614 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.614 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.614 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.614 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:11:17.614 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.614 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.615 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.615 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.615 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.615 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.615 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.615 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:11:17.615 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.615 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.615 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.615 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.615 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.615 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.615 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.616 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:11:17.616 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:16.899700+0000" 2024-09-22T03:11:17.616 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.616 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.616 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.616 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.616 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.616 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.616 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:11:17.616 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.616 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.616 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.616 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.616 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:16.902875+0000" 2024-09-22T03:11:17.617 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.617 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.617 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:11:17.617 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:16.919309+0000" 2024-09-22T03:11:17.617 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.617 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.617 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.617 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.617 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.617 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.617 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:11:17.617 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.617 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.617 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.617 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.618 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.618 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.618 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.618 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:11:17.618 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:16.946691+0000" 2024-09-22T03:11:17.618 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.618 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.618 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.618 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.618 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.618 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.618 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:11:17.618 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.618 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:11:17.619 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:11:17.619 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.619 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-22T03:11:17.619 INFO:teuthology.orchestra.run.smithi196.stdout: "type": "client_request", 2024-09-22T03:11:17.619 INFO:teuthology.orchestra.run.smithi196.stdout: "id": 436, 2024-09-22T03:11:17.619 INFO:teuthology.orchestra.run.smithi196.stdout: "detail": { 2024-09-22T03:11:17.619 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.619 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:11:17.619 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:16.893446+0000" 2024-09-22T03:11:17.619 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.620 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.620 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:11:17.620 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:16.893449+0000" 2024-09-22T03:11:17.620 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.620 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.620 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.620 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.620 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.620 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.620 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:11:17.620 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:16.893537+0000" 2024-09-22T03:11:17.620 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.620 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.620 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.621 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.621 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.621 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.621 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:11:17.621 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.621 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.621 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.621 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.621 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.621 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.621 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.622 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:11:17.622 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:16.893587+0000" 2024-09-22T03:11:17.622 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.622 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.622 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.622 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.622 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.622 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.622 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:11:17.622 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:16.894701+0000" 2024-09-22T03:11:17.622 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.623 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.623 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.623 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.623 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.623 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.623 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:11:17.623 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.623 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.623 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.623 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.623 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.624 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.624 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.624 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:11:17.624 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:16.951953+0000" 2024-09-22T03:11:17.624 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.624 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.624 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:11:17.624 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:16.898383+0000" 2024-09-22T03:11:17.624 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.624 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.624 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.625 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.625 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.625 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.625 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:11:17.625 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.625 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.625 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.625 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.625 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.625 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.625 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.625 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:11:17.626 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:16.898757+0000" 2024-09-22T03:11:17.626 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.626 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.626 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.626 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.626 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.626 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.626 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:11:17.626 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.626 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.626 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.627 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.627 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.627 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.627 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.627 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:11:17.627 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:16.899654+0000" 2024-09-22T03:11:17.627 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.628 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.628 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.628 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.628 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.628 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.628 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:11:17.628 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.629 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.629 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.629 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.629 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.629 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.629 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.629 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:11:17.629 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.630 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.630 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.630 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.630 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.630 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.630 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.630 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:11:17.630 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.630 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.631 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.631 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.631 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.631 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.631 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.631 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:11:17.631 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:16.900489+0000" 2024-09-22T03:11:17.631 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.631 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.631 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.631 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.631 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.632 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.632 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:11:17.632 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.632 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.632 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.632 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.632 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:16.906166+0000" 2024-09-22T03:11:17.632 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.632 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.632 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:11:17.632 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:16.923632+0000" 2024-09-22T03:11:17.633 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.633 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.633 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.633 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.633 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.633 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.633 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:11:17.633 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.633 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.633 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.634 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.634 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.634 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.634 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.634 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:11:17.634 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:16.950116+0000" 2024-09-22T03:11:17.634 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.634 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.634 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.634 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.634 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.635 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.635 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:11:17.635 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.635 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:11:17.635 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:11:17.635 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.635 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-22T03:11:17.635 INFO:teuthology.orchestra.run.smithi196.stdout: "type": "client_request", 2024-09-22T03:11:17.635 INFO:teuthology.orchestra.run.smithi196.stdout: "id": 437, 2024-09-22T03:11:17.635 INFO:teuthology.orchestra.run.smithi196.stdout: "detail": { 2024-09-22T03:11:17.635 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.635 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:11:17.636 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:16.893761+0000" 2024-09-22T03:11:17.636 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.636 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.636 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:11:17.636 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:16.893764+0000" 2024-09-22T03:11:17.636 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.636 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.636 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.636 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.636 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.636 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.637 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:11:17.637 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:16.893829+0000" 2024-09-22T03:11:17.637 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.637 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.637 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.637 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.637 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.637 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.637 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:11:17.637 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.637 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.637 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.638 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.638 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.638 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.638 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.638 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:11:17.638 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:16.893890+0000" 2024-09-22T03:11:17.638 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.638 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.638 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.638 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.638 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.639 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.639 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:11:17.639 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:16.894788+0000" 2024-09-22T03:11:17.639 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.639 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.639 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.639 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.639 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.640 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.640 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:11:17.640 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.640 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.640 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.640 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.640 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.640 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.640 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.640 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:11:17.640 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:16.971600+0000" 2024-09-22T03:11:17.640 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.641 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.641 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:11:17.641 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:16.899148+0000" 2024-09-22T03:11:17.641 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.641 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.641 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.641 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.641 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.641 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.641 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:11:17.641 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.641 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.641 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.642 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.642 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.642 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.642 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.642 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:11:17.642 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:16.899602+0000" 2024-09-22T03:11:17.642 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.642 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.642 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.642 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.643 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.643 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.643 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:11:17.643 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.643 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.643 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.643 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.643 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.643 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.643 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.643 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:11:17.643 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:16.900573+0000" 2024-09-22T03:11:17.644 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.644 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.644 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.644 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.644 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.644 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.644 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:11:17.644 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.644 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.644 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.644 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.644 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.644 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.644 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.645 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:11:17.645 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.645 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.645 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.645 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.645 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.645 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.645 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.645 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:11:17.645 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.645 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.645 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.645 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.645 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.646 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.646 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.646 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:11:17.646 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:16.901301+0000" 2024-09-22T03:11:17.646 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.646 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.646 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.646 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.646 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.646 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.646 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:11:17.646 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.646 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.646 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.647 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.647 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.647 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.647 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.647 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:11:17.647 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:16.950945+0000" 2024-09-22T03:11:17.647 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.647 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.647 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.647 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.647 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.647 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.647 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:11:17.648 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.648 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.648 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.648 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.648 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.648 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.648 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.648 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:11:17.648 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:16.971304+0000" 2024-09-22T03:11:17.648 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.648 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.648 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.648 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.648 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.648 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.649 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:11:17.649 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.649 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:11:17.649 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:11:17.649 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.649 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-22T03:11:17.649 INFO:teuthology.orchestra.run.smithi196.stdout: "type": "client_request", 2024-09-22T03:11:17.649 INFO:teuthology.orchestra.run.smithi196.stdout: "id": 438, 2024-09-22T03:11:17.649 INFO:teuthology.orchestra.run.smithi196.stdout: "detail": { 2024-09-22T03:11:17.649 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.649 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:11:17.649 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:16.894234+0000" 2024-09-22T03:11:17.649 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.650 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.650 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:11:17.650 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:16.894238+0000" 2024-09-22T03:11:17.650 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.650 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.650 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.650 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.650 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.650 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.650 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:11:17.650 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:16.894301+0000" 2024-09-22T03:11:17.650 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.650 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.650 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.651 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.651 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.651 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.651 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:11:17.651 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.651 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.651 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.651 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.651 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.651 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.651 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.651 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:11:17.651 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:16.894341+0000" 2024-09-22T03:11:17.652 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.652 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.652 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.652 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.652 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.652 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.652 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:11:17.652 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:16.895580+0000" 2024-09-22T03:11:17.652 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.652 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.652 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.652 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.652 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.652 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.653 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:11:17.653 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.653 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.653 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.653 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.653 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.653 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.653 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.653 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:11:17.653 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:16.980655+0000" 2024-09-22T03:11:17.653 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.653 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.653 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:11:17.653 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:16.900316+0000" 2024-09-22T03:11:17.653 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.654 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.654 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.654 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.654 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.654 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.654 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:11:17.654 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.654 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.654 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.654 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.654 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.654 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.655 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.655 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:11:17.655 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:16.900724+0000" 2024-09-22T03:11:17.655 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.655 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.655 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.655 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.655 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.655 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.655 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:11:17.655 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.655 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.655 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.655 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.656 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.656 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.656 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.656 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:11:17.656 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:16.901472+0000" 2024-09-22T03:11:17.656 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.656 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.656 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.656 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.656 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.656 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.656 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:11:17.656 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.656 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.657 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.657 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.657 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.657 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.657 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.657 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:11:17.657 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.657 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.657 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.657 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.657 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.657 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.657 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.657 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:11:17.658 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.658 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.658 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.658 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.658 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.658 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.658 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.658 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:11:17.658 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:16.902096+0000" 2024-09-22T03:11:17.658 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.658 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.658 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.658 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.658 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.659 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.659 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:11:17.659 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.659 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.659 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.659 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.659 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:16.951217+0000" 2024-09-22T03:11:17.659 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.659 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.659 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:11:17.659 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:16.951756+0000" 2024-09-22T03:11:17.659 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.659 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.660 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.660 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.660 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.660 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.660 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:11:17.660 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.660 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.660 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.660 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.660 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.660 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.660 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.660 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:11:17.660 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:16.980222+0000" 2024-09-22T03:11:17.661 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.661 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.661 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.661 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.661 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.661 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.661 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:11:17.661 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.661 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:11:17.661 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:11:17.661 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.661 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-22T03:11:17.661 INFO:teuthology.orchestra.run.smithi196.stdout: "type": "client_request", 2024-09-22T03:11:17.661 INFO:teuthology.orchestra.run.smithi196.stdout: "id": 439, 2024-09-22T03:11:17.662 INFO:teuthology.orchestra.run.smithi196.stdout: "detail": { 2024-09-22T03:11:17.662 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.662 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:11:17.662 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:16.895824+0000" 2024-09-22T03:11:17.662 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.662 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.662 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:11:17.662 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:16.895828+0000" 2024-09-22T03:11:17.662 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.662 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.662 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.662 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.662 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.663 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.663 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:11:17.663 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:16.895892+0000" 2024-09-22T03:11:17.663 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.663 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.663 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.663 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.663 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.663 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.663 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:11:17.663 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.663 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.663 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.663 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.664 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.664 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.664 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.664 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:11:17.664 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:16.895934+0000" 2024-09-22T03:11:17.664 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.664 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.664 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.664 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.664 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.664 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.664 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:11:17.664 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:16.896428+0000" 2024-09-22T03:11:17.664 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.665 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.665 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.665 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.665 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.665 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.665 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:11:17.665 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.665 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.665 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.665 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.665 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.665 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.665 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.665 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:11:17.666 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:16.993430+0000" 2024-09-22T03:11:17.666 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.666 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.666 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:11:17.666 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:16.901215+0000" 2024-09-22T03:11:17.666 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.666 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.666 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.666 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.666 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.666 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.666 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:11:17.666 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.667 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.667 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.667 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.667 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.667 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.667 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.667 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:11:17.667 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:16.901752+0000" 2024-09-22T03:11:17.667 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.667 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.667 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.668 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.668 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.668 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.668 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:11:17.668 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.668 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.668 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.668 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.668 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.668 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.668 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.668 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:11:17.668 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:16.902781+0000" 2024-09-22T03:11:17.669 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.669 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.669 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.669 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.669 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.669 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.669 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:11:17.669 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.669 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.669 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.669 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.669 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.669 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.669 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.670 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:11:17.670 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.670 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.670 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.670 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.670 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.670 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.670 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.670 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:11:17.670 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.670 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.670 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.670 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.671 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.671 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.671 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.671 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:11:17.671 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:16.904203+0000" 2024-09-22T03:11:17.671 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.671 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.671 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.671 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.671 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.671 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.671 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:11:17.671 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.671 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.672 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.672 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.672 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:16.950842+0000" 2024-09-22T03:11:17.672 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.672 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.672 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:11:17.672 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:16.954874+0000" 2024-09-22T03:11:17.672 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.672 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.672 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.672 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.672 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.672 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.672 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:11:17.673 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.673 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.673 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.673 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.673 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.673 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.673 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.673 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:11:17.673 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:16.993046+0000" 2024-09-22T03:11:17.673 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.673 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.673 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.673 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.673 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.673 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.674 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:11:17.674 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.674 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:11:17.674 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:11:17.674 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.674 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-22T03:11:17.674 INFO:teuthology.orchestra.run.smithi196.stdout: "type": "client_request", 2024-09-22T03:11:17.674 INFO:teuthology.orchestra.run.smithi196.stdout: "id": 440, 2024-09-22T03:11:17.674 INFO:teuthology.orchestra.run.smithi196.stdout: "detail": { 2024-09-22T03:11:17.674 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.674 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:11:17.674 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:16.896794+0000" 2024-09-22T03:11:17.674 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.674 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.675 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:11:17.675 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:16.896797+0000" 2024-09-22T03:11:17.675 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.675 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.675 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.675 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.675 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.675 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.675 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:11:17.675 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:16.896848+0000" 2024-09-22T03:11:17.675 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.675 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.675 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.675 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.676 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.676 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.676 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:11:17.676 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.676 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.676 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.676 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.676 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.676 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.676 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.676 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:11:17.676 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:16.896890+0000" 2024-09-22T03:11:17.676 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.677 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.677 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.677 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.677 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.677 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.677 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:11:17.677 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:16.897673+0000" 2024-09-22T03:11:17.677 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.677 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.677 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.677 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.677 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.677 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.677 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:11:17.678 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.678 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.678 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.678 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.678 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.678 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.678 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.678 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:11:17.678 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:17.003313+0000" 2024-09-22T03:11:17.678 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.678 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.678 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:11:17.678 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:16.901964+0000" 2024-09-22T03:11:17.679 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.679 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.679 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.679 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.679 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.679 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.679 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:11:17.679 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.679 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.679 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.679 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.679 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.679 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.679 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.680 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:11:17.680 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:16.903062+0000" 2024-09-22T03:11:17.680 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.680 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.680 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.680 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.680 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.680 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.680 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:11:17.680 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.680 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.680 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.680 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.680 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.681 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.681 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.681 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:11:17.681 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:16.904284+0000" 2024-09-22T03:11:17.681 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.681 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.681 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.681 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.681 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.681 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.681 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:11:17.681 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.681 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.681 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.682 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.682 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.682 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.682 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.682 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:11:17.682 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.682 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.682 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.682 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.682 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.682 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.682 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.682 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:11:17.682 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.683 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.683 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.683 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.683 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.683 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.683 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.683 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:11:17.683 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:16.906443+0000" 2024-09-22T03:11:17.683 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.683 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.683 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.683 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.683 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.684 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.684 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:11:17.684 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.684 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.684 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.684 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.684 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:16.950642+0000" 2024-09-22T03:11:17.684 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.684 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.684 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:11:17.684 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:16.957888+0000" 2024-09-22T03:11:17.684 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.684 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.685 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.685 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.685 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.685 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.685 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:11:17.685 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.685 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.685 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.685 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.685 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.685 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.685 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.685 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:11:17.685 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:17.002927+0000" 2024-09-22T03:11:17.685 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.686 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.686 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.686 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.686 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.686 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.686 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:11:17.686 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.686 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:11:17.686 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:11:17.686 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.686 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-22T03:11:17.686 INFO:teuthology.orchestra.run.smithi196.stdout: "type": "client_request", 2024-09-22T03:11:17.686 INFO:teuthology.orchestra.run.smithi196.stdout: "id": 441, 2024-09-22T03:11:17.687 INFO:teuthology.orchestra.run.smithi196.stdout: "detail": { 2024-09-22T03:11:17.687 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.687 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:11:17.687 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:16.899058+0000" 2024-09-22T03:11:17.687 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.687 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.687 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:11:17.687 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:16.899061+0000" 2024-09-22T03:11:17.687 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.687 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.687 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.687 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.687 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.687 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.688 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:11:17.688 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:16.899143+0000" 2024-09-22T03:11:17.688 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.688 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.688 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.688 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.688 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.688 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.688 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:11:17.688 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.688 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.688 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.688 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.688 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.688 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.689 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.689 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:11:17.689 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:16.899202+0000" 2024-09-22T03:11:17.689 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.689 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.689 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.689 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.689 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.689 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.689 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:11:17.689 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:16.899728+0000" 2024-09-22T03:11:17.689 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.689 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.689 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.690 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.690 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.690 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.690 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:11:17.690 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.690 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.690 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.690 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.690 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.690 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.690 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.690 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:11:17.690 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:17.009155+0000" 2024-09-22T03:11:17.690 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.691 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.691 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:11:17.691 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:16.904160+0000" 2024-09-22T03:11:17.691 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.691 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.691 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.691 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.691 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.691 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.691 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:11:17.691 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.691 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.691 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.692 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.692 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.692 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.692 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.692 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:11:17.692 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:16.904411+0000" 2024-09-22T03:11:17.692 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.692 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.692 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.692 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.692 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.692 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.692 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:11:17.693 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.693 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.693 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.693 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.693 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.693 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.693 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.693 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:11:17.693 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:16.906757+0000" 2024-09-22T03:11:17.693 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.693 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.693 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.693 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.693 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.693 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.694 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:11:17.694 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.694 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.694 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.694 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.694 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.694 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.694 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.694 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:11:17.694 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.694 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.694 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.694 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.694 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.694 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.695 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.695 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:11:17.695 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.695 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.695 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.695 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.695 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.695 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.695 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.695 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:11:17.695 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:16.906943+0000" 2024-09-22T03:11:17.695 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.695 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.695 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.696 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.696 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.696 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.696 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:11:17.696 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.696 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.696 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.696 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.696 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:16.951316+0000" 2024-09-22T03:11:17.696 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.696 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.696 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:11:17.696 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:16.959594+0000" 2024-09-22T03:11:17.697 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.697 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.697 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.697 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.697 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.697 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.697 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:11:17.697 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.697 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.697 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.697 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.697 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.697 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.697 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.698 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:11:17.698 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:17.008930+0000" 2024-09-22T03:11:17.698 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.698 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.698 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.698 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.698 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.698 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.698 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:11:17.698 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.698 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:11:17.698 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:11:17.698 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.698 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-22T03:11:17.699 INFO:teuthology.orchestra.run.smithi196.stdout: "type": "client_request", 2024-09-22T03:11:17.699 INFO:teuthology.orchestra.run.smithi196.stdout: "id": 442, 2024-09-22T03:11:17.699 INFO:teuthology.orchestra.run.smithi196.stdout: "detail": { 2024-09-22T03:11:17.699 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.699 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:11:17.699 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:16.900027+0000" 2024-09-22T03:11:17.699 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.699 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.699 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:11:17.699 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:16.900030+0000" 2024-09-22T03:11:17.699 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.699 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.699 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.700 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.700 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.700 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.700 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:11:17.700 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:16.900123+0000" 2024-09-22T03:11:17.700 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.700 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.700 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.700 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.700 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.700 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.700 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:11:17.700 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.700 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.701 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.701 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.701 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.701 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.701 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.701 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:11:17.701 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:16.900168+0000" 2024-09-22T03:11:17.701 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.701 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.701 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.701 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.701 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.701 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.702 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:11:17.702 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:16.900777+0000" 2024-09-22T03:11:17.702 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.702 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.702 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.702 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.702 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.702 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.702 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:11:17.702 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.702 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.702 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.702 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.702 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.703 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.703 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.703 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:11:17.703 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:17.013643+0000" 2024-09-22T03:11:17.703 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.703 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.703 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:11:17.703 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:16.905902+0000" 2024-09-22T03:11:17.703 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.703 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.703 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.703 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.703 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.703 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.704 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:11:17.704 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.704 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.704 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.704 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.704 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.704 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.704 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.704 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:11:17.704 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:16.906805+0000" 2024-09-22T03:11:17.704 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.704 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.704 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.705 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.705 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.705 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.705 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:11:17.705 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.705 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.705 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.705 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.705 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.705 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.705 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.705 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:11:17.705 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:16.907290+0000" 2024-09-22T03:11:17.706 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.706 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.706 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.706 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.706 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.706 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.706 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:11:17.706 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.706 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.706 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.706 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.706 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.706 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.707 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.707 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:11:17.707 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.707 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.707 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.707 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.707 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.707 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.707 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.707 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:11:17.707 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.707 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.707 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.707 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.708 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.708 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.708 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.708 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:11:17.708 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:16.908020+0000" 2024-09-22T03:11:17.708 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.708 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.708 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.708 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.708 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.708 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.708 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:11:17.708 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.709 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.709 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.709 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.709 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:16.951111+0000" 2024-09-22T03:11:17.709 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.709 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.709 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:11:17.709 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:16.961273+0000" 2024-09-22T03:11:17.709 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.709 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.709 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.709 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.709 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.709 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.710 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:11:17.710 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.710 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.710 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.710 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.710 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.710 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.710 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.710 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:11:17.710 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:17.013392+0000" 2024-09-22T03:11:17.710 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.710 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.710 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.710 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.711 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.711 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.711 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:11:17.711 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.711 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:11:17.711 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:11:17.711 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.711 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-22T03:11:17.711 INFO:teuthology.orchestra.run.smithi196.stdout: "type": "client_request", 2024-09-22T03:11:17.711 INFO:teuthology.orchestra.run.smithi196.stdout: "id": 443, 2024-09-22T03:11:17.711 INFO:teuthology.orchestra.run.smithi196.stdout: "detail": { 2024-09-22T03:11:17.711 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.711 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:11:17.711 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:16.900673+0000" 2024-09-22T03:11:17.712 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.712 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.712 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:11:17.712 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:16.900676+0000" 2024-09-22T03:11:17.712 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.712 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.712 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.712 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.712 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.712 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.712 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:11:17.712 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:16.900726+0000" 2024-09-22T03:11:17.712 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.713 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.713 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.713 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.713 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.713 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.713 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:11:17.713 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.713 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.713 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.713 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.713 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.713 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.713 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.713 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:11:17.714 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:16.900780+0000" 2024-09-22T03:11:17.714 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.714 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.714 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.714 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.714 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.714 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.714 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:11:17.714 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:16.902738+0000" 2024-09-22T03:11:17.714 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.714 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.714 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.714 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.715 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.715 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.715 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:11:17.715 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.715 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.715 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.715 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.715 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.715 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.715 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.715 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:11:17.715 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:17.017094+0000" 2024-09-22T03:11:17.715 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.715 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.716 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:11:17.716 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:16.907144+0000" 2024-09-22T03:11:17.716 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.716 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.716 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.716 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.716 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.716 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.716 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:11:17.716 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.716 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.716 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.716 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.716 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.716 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.716 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.717 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:11:17.717 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:16.907439+0000" 2024-09-22T03:11:17.717 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.717 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.717 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.717 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.717 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.717 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.717 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:11:17.717 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.717 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.717 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.717 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.717 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.717 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.718 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.718 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:11:17.718 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:16.908108+0000" 2024-09-22T03:11:17.718 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.718 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.718 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.718 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.718 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.718 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.718 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:11:17.718 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.718 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.718 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.718 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.719 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.719 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.719 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.719 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:11:17.719 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.719 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.719 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.719 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.719 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.719 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.719 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.719 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:11:17.719 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.719 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.719 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.720 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.720 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.720 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.720 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.720 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:11:17.720 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:16.908687+0000" 2024-09-22T03:11:17.720 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.720 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.720 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.720 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.720 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.720 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.720 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:11:17.720 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.721 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.721 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.721 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.721 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:16.950931+0000" 2024-09-22T03:11:17.721 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.721 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.721 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:11:17.721 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:16.963649+0000" 2024-09-22T03:11:17.721 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.721 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.721 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.721 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.721 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.722 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.722 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:11:17.722 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.722 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.722 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.722 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.722 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.722 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.722 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.722 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:11:17.722 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:17.016809+0000" 2024-09-22T03:11:17.722 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.722 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.723 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.723 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.723 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.723 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.723 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:11:17.723 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.723 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:11:17.723 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:11:17.723 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.723 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-22T03:11:17.723 INFO:teuthology.orchestra.run.smithi196.stdout: "type": "client_request", 2024-09-22T03:11:17.723 INFO:teuthology.orchestra.run.smithi196.stdout: "id": 468, 2024-09-22T03:11:17.723 INFO:teuthology.orchestra.run.smithi196.stdout: "detail": { 2024-09-22T03:11:17.723 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.724 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:11:17.724 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:17.072565+0000" 2024-09-22T03:11:17.724 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.724 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.724 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:11:17.724 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:17.072578+0000" 2024-09-22T03:11:17.724 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.724 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.724 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.724 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.724 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.724 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.724 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:11:17.724 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:17.072704+0000" 2024-09-22T03:11:17.725 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.725 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.725 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.725 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.725 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.725 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.725 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:11:17.725 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.725 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.725 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.725 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.725 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.725 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.725 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.725 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:11:17.726 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:17.072786+0000" 2024-09-22T03:11:17.726 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.726 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.726 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.726 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.726 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.726 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.726 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:11:17.726 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:17.072989+0000" 2024-09-22T03:11:17.726 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.726 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.726 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.726 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.727 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.727 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.727 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:11:17.727 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.727 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.727 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.727 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.727 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.727 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.727 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.727 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:11:17.727 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:17.100327+0000" 2024-09-22T03:11:17.727 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.727 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.727 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:11:17.728 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:17.073350+0000" 2024-09-22T03:11:17.728 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.728 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.728 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.728 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.728 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.728 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.728 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:11:17.728 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.728 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.728 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.728 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.728 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.728 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.728 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.729 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:11:17.729 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:17.073537+0000" 2024-09-22T03:11:17.729 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.729 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.729 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.729 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.729 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.729 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.729 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:11:17.729 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.729 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.729 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.729 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.729 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.730 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.730 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.730 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:11:17.730 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:17.073659+0000" 2024-09-22T03:11:17.730 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.730 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.730 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.730 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.730 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.730 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.730 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:11:17.730 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.730 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.730 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.731 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.731 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.731 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.731 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.731 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:11:17.731 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.731 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.731 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.731 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.731 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.731 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.731 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.731 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:11:17.731 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.732 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.732 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.732 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.732 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.732 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.732 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.732 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:11:17.732 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:17.073799+0000" 2024-09-22T03:11:17.732 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.732 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.732 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.732 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.732 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.732 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.732 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:11:17.733 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.733 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.733 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.733 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.733 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.733 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.733 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.733 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:11:17.733 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:17.074602+0000" 2024-09-22T03:11:17.733 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.733 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.733 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.733 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.733 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.734 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.734 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:11:17.734 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.734 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.734 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.734 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.734 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.734 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.734 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.734 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:11:17.734 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:17.099870+0000" 2024-09-22T03:11:17.734 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.734 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.735 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.735 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.735 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.735 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.735 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:11:17.735 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.735 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:11:17.735 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:11:17.735 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.735 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-22T03:11:17.735 INFO:teuthology.orchestra.run.smithi196.stdout: "type": "client_request", 2024-09-22T03:11:17.735 INFO:teuthology.orchestra.run.smithi196.stdout: "id": 469, 2024-09-22T03:11:17.735 INFO:teuthology.orchestra.run.smithi196.stdout: "detail": { 2024-09-22T03:11:17.735 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.736 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:11:17.736 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:17.075185+0000" 2024-09-22T03:11:17.736 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.736 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.736 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:11:17.736 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:17.075191+0000" 2024-09-22T03:11:17.736 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.736 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.736 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.736 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.736 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.736 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.736 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:11:17.736 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:17.075263+0000" 2024-09-22T03:11:17.737 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.737 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.737 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.737 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.737 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.737 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.737 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:11:17.737 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.737 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.737 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.737 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.737 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.737 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.737 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.738 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:11:17.738 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:17.075327+0000" 2024-09-22T03:11:17.738 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.738 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.738 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.738 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.738 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.738 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.738 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:11:17.738 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:17.077454+0000" 2024-09-22T03:11:17.738 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.738 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.738 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.738 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.739 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.739 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.739 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:11:17.739 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.739 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.739 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.739 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.739 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.739 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.739 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.739 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:11:17.739 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:17.112068+0000" 2024-09-22T03:11:17.739 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.739 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.740 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:11:17.740 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:17.077719+0000" 2024-09-22T03:11:17.740 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.740 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.740 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.740 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.740 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.740 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.740 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:11:17.740 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.740 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.740 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.740 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.740 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.740 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.741 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.741 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:11:17.741 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:17.077824+0000" 2024-09-22T03:11:17.741 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.741 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.741 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.741 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.741 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.741 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.741 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:11:17.741 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.741 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.741 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.742 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.742 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.742 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.742 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.742 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:11:17.742 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:17.078085+0000" 2024-09-22T03:11:17.742 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.742 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.742 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.742 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.742 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.742 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.742 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:11:17.743 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.743 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.743 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.743 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.743 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.743 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.743 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.743 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:11:17.743 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.743 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.743 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.743 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.743 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.743 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.744 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.744 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:11:17.744 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.744 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.744 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.744 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.744 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.744 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.744 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.744 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:11:17.744 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:17.078265+0000" 2024-09-22T03:11:17.744 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.744 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.744 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.745 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.745 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.745 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.745 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:11:17.745 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.745 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.745 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.745 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.745 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.745 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.745 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.745 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:11:17.745 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:17.079017+0000" 2024-09-22T03:11:17.745 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.745 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.746 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.746 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.746 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.746 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.746 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:11:17.746 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.746 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.746 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.746 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.746 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.746 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.746 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.746 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:11:17.747 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:17.111816+0000" 2024-09-22T03:11:17.747 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.747 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.747 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.747 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.747 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.747 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.747 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:11:17.747 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.747 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:11:17.747 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:11:17.747 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.747 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-22T03:11:17.747 INFO:teuthology.orchestra.run.smithi196.stdout: "type": "client_request", 2024-09-22T03:11:17.748 INFO:teuthology.orchestra.run.smithi196.stdout: "id": 470, 2024-09-22T03:11:17.748 INFO:teuthology.orchestra.run.smithi196.stdout: "detail": { 2024-09-22T03:11:17.748 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.748 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:11:17.748 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:17.075553+0000" 2024-09-22T03:11:17.748 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.748 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.748 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:11:17.748 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:17.075556+0000" 2024-09-22T03:11:17.748 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.748 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.748 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.748 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.748 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.749 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.749 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:11:17.749 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:17.075655+0000" 2024-09-22T03:11:17.749 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.749 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.749 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.749 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.749 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.749 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.749 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:11:17.749 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.749 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.749 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.749 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.750 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.750 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.750 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.750 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:11:17.750 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:17.075744+0000" 2024-09-22T03:11:17.750 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.750 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.750 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.750 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.750 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.750 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.750 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:11:17.750 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:17.077500+0000" 2024-09-22T03:11:17.750 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.751 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.751 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.751 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.751 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.751 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.751 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:11:17.751 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.751 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.751 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.751 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.751 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.751 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.752 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.752 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:11:17.752 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:17.118833+0000" 2024-09-22T03:11:17.752 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.752 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.752 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:11:17.752 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:17.077986+0000" 2024-09-22T03:11:17.752 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.752 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.752 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.752 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.752 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.752 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.752 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:11:17.753 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.753 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.753 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.753 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.753 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.753 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.753 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.753 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:11:17.753 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:17.078310+0000" 2024-09-22T03:11:17.753 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.753 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.753 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.753 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.753 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.754 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.754 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:11:17.754 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.754 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.754 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.754 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.754 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.754 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.754 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.754 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:11:17.754 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:17.078499+0000" 2024-09-22T03:11:17.754 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.754 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.754 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.755 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.755 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.755 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.755 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:11:17.755 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.755 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.755 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.755 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.755 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.755 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.755 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.755 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:11:17.755 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.755 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.756 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.756 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.756 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.756 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.756 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.756 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:11:17.756 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.756 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.756 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.756 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.756 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.756 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.756 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.756 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:11:17.757 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:17.079240+0000" 2024-09-22T03:11:17.757 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.757 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.757 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.757 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.757 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.757 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.757 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:11:17.757 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.757 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.757 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.757 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.757 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:17.079638+0000" 2024-09-22T03:11:17.758 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.758 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.758 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:11:17.758 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:17.080156+0000" 2024-09-22T03:11:17.758 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.758 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.758 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.758 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.758 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.758 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.758 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:11:17.758 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.758 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.759 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.759 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.759 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.759 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.759 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.759 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:11:17.759 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:17.118462+0000" 2024-09-22T03:11:17.759 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.759 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.759 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.759 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.759 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.759 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.759 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:11:17.760 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.760 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:11:17.760 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:11:17.760 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.760 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-22T03:11:17.760 INFO:teuthology.orchestra.run.smithi196.stdout: "type": "client_request", 2024-09-22T03:11:17.760 INFO:teuthology.orchestra.run.smithi196.stdout: "id": 471, 2024-09-22T03:11:17.760 INFO:teuthology.orchestra.run.smithi196.stdout: "detail": { 2024-09-22T03:11:17.760 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.760 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:11:17.760 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:17.077283+0000" 2024-09-22T03:11:17.760 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.760 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.761 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:11:17.761 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:17.077288+0000" 2024-09-22T03:11:17.761 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.761 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.761 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.761 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.761 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.761 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.761 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:11:17.761 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:17.077388+0000" 2024-09-22T03:11:17.761 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.761 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.761 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.761 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.762 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.762 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.762 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:11:17.762 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.762 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.762 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.762 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.762 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.762 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.762 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.762 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:11:17.762 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:17.077476+0000" 2024-09-22T03:11:17.762 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.763 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.763 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.763 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.763 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.763 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.763 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:11:17.763 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:17.079738+0000" 2024-09-22T03:11:17.763 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.763 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.763 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.763 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.763 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.763 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.763 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:11:17.763 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.764 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.764 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.764 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.764 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.764 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.764 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.764 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:11:17.764 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:17.119335+0000" 2024-09-22T03:11:17.764 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.764 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.764 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:11:17.764 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:17.080466+0000" 2024-09-22T03:11:17.764 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.764 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.764 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.765 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.765 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.765 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.765 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:11:17.765 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.765 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.765 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.765 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.765 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.765 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.765 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.765 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:11:17.765 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:17.080758+0000" 2024-09-22T03:11:17.766 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.766 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.766 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.766 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.766 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.766 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.766 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:11:17.766 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.766 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.766 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.766 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.766 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.766 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.766 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.766 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:11:17.767 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:17.082560+0000" 2024-09-22T03:11:17.767 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.767 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.767 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.767 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.767 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.767 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.767 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:11:17.767 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.767 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.767 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.767 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.767 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.768 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.768 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.768 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:11:17.768 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.768 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.768 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.768 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.768 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.768 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.768 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.768 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:11:17.768 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.768 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.768 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.769 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.769 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.769 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.769 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.769 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:11:17.769 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:17.083292+0000" 2024-09-22T03:11:17.769 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.769 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.769 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.769 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.769 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.769 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.769 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:11:17.770 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.770 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.770 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.770 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.770 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:17.084085+0000" 2024-09-22T03:11:17.770 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.770 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.770 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:11:17.770 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:17.085893+0000" 2024-09-22T03:11:17.770 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.770 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.770 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.770 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.770 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.771 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.771 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:11:17.771 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.771 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.771 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.771 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.771 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.771 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.771 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.771 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:11:17.771 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:17.119121+0000" 2024-09-22T03:11:17.771 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.771 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.772 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.772 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.772 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.772 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.772 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:11:17.772 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.772 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:11:17.772 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:11:17.772 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.772 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-22T03:11:17.772 INFO:teuthology.orchestra.run.smithi196.stdout: "type": "client_request", 2024-09-22T03:11:17.772 INFO:teuthology.orchestra.run.smithi196.stdout: "id": 473, 2024-09-22T03:11:17.772 INFO:teuthology.orchestra.run.smithi196.stdout: "detail": { 2024-09-22T03:11:17.772 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.773 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:11:17.773 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:17.080187+0000" 2024-09-22T03:11:17.773 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.773 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.773 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:11:17.773 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:17.080193+0000" 2024-09-22T03:11:17.773 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.773 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.773 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.773 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.773 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.773 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.773 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:11:17.774 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:17.080276+0000" 2024-09-22T03:11:17.774 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.774 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.774 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.774 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.774 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.774 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.774 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:11:17.774 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.774 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.774 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.774 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.774 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.774 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.775 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.775 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:11:17.775 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:17.080329+0000" 2024-09-22T03:11:17.775 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.775 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.775 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.775 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.775 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.775 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.775 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:11:17.775 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:17.082344+0000" 2024-09-22T03:11:17.775 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.775 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.776 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.776 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.776 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.776 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.776 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:11:17.776 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.776 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.776 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.776 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.776 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.776 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.776 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.776 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:11:17.776 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:17.122745+0000" 2024-09-22T03:11:17.777 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.777 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.777 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:11:17.777 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:17.082886+0000" 2024-09-22T03:11:17.777 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.777 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.777 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.777 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.777 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.777 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.777 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:11:17.777 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.777 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.777 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.778 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.778 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.778 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.778 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.778 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:11:17.778 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:17.083125+0000" 2024-09-22T03:11:17.778 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.778 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.778 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.778 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.778 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.778 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.778 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:11:17.779 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.779 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.779 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.779 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.779 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.779 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.779 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.779 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:11:17.779 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:17.084189+0000" 2024-09-22T03:11:17.779 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.779 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.779 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.779 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.779 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.780 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.780 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:11:17.780 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.780 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.780 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.780 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.780 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.780 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.780 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.780 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:11:17.780 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.780 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.780 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.781 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.781 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.781 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.781 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.781 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:11:17.781 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.781 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.781 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.781 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.781 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.781 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.781 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.781 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:11:17.781 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:17.084430+0000" 2024-09-22T03:11:17.782 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.782 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.782 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.782 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.782 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.782 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.782 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:11:17.782 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.782 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.782 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.782 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.782 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:17.085063+0000" 2024-09-22T03:11:17.782 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.782 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.783 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:11:17.783 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:17.089903+0000" 2024-09-22T03:11:17.783 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.783 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.783 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.783 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.783 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.783 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.783 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:11:17.783 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.783 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.783 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.783 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.783 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.784 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.784 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.784 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:11:17.784 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:17.122517+0000" 2024-09-22T03:11:17.784 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.784 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.784 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.784 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.784 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.784 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.784 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:11:17.784 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.784 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:11:17.784 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:11:17.784 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.785 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-22T03:11:17.785 INFO:teuthology.orchestra.run.smithi196.stdout: "type": "client_request", 2024-09-22T03:11:17.785 INFO:teuthology.orchestra.run.smithi196.stdout: "id": 474, 2024-09-22T03:11:17.785 INFO:teuthology.orchestra.run.smithi196.stdout: "detail": { 2024-09-22T03:11:17.785 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.785 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:11:17.785 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:17.080803+0000" 2024-09-22T03:11:17.785 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.785 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.785 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:11:17.785 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:17.080809+0000" 2024-09-22T03:11:17.785 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.785 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.785 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.786 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.786 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.786 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.786 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:11:17.786 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:17.080918+0000" 2024-09-22T03:11:17.786 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.786 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.786 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.786 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.786 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.786 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.786 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:11:17.786 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.786 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.787 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.787 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.787 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.787 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.787 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.787 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:11:17.787 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:17.080990+0000" 2024-09-22T03:11:17.787 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.787 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.787 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.787 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.787 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.787 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.787 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:11:17.788 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:17.082462+0000" 2024-09-22T03:11:17.788 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.788 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.788 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.788 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.788 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.788 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.788 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:11:17.788 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.788 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.788 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.788 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.788 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.788 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.789 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.789 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:11:17.789 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:17.126333+0000" 2024-09-22T03:11:17.789 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.789 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.789 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:11:17.789 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:17.083597+0000" 2024-09-22T03:11:17.789 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.789 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.789 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.789 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.789 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.789 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.789 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:11:17.790 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.790 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.790 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.790 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.790 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.790 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.790 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.790 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:11:17.790 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:17.084578+0000" 2024-09-22T03:11:17.790 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.790 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.790 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.790 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.790 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.791 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.791 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:11:17.791 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.791 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.791 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.791 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.791 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.791 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.791 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.791 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:11:17.791 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:17.085240+0000" 2024-09-22T03:11:17.791 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.791 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.791 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.792 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.792 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.792 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.792 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:11:17.792 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.792 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.792 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.792 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.792 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.792 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.792 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.792 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:11:17.792 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.792 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.793 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.793 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.793 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.793 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.793 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.793 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:11:17.793 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.793 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.793 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.793 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.793 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.793 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.793 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.793 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:11:17.794 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:17.085365+0000" 2024-09-22T03:11:17.794 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.794 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.794 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.794 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.794 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.794 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.794 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:11:17.794 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.794 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.794 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.794 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.794 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:17.085599+0000" 2024-09-22T03:11:17.794 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.795 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.795 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:11:17.795 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:17.094396+0000" 2024-09-22T03:11:17.795 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.795 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.795 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.795 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.795 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.795 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.795 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:11:17.795 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.795 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.795 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.795 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.796 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.796 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.796 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.796 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:11:17.796 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:17.125927+0000" 2024-09-22T03:11:17.796 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.796 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.796 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.796 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.796 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.796 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.796 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:11:17.796 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.796 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:11:17.797 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:11:17.797 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.797 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-22T03:11:17.797 INFO:teuthology.orchestra.run.smithi196.stdout: "type": "client_request", 2024-09-22T03:11:17.797 INFO:teuthology.orchestra.run.smithi196.stdout: "id": 475, 2024-09-22T03:11:17.797 INFO:teuthology.orchestra.run.smithi196.stdout: "detail": { 2024-09-22T03:11:17.797 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.797 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:11:17.797 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:17.084552+0000" 2024-09-22T03:11:17.797 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.797 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.797 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:11:17.797 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:17.084557+0000" 2024-09-22T03:11:17.797 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.798 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.798 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.798 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.798 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.798 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.798 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:11:17.798 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:17.084648+0000" 2024-09-22T03:11:17.798 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.798 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.798 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.798 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.798 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.798 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.798 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:11:17.799 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.799 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.799 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.799 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.799 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.799 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.799 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.799 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:11:17.799 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:17.084844+0000" 2024-09-22T03:11:17.799 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.799 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.799 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.799 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.799 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.800 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.800 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:11:17.800 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:17.085667+0000" 2024-09-22T03:11:17.800 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.800 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.800 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.800 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.800 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.800 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.800 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:11:17.800 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.800 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.800 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.800 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.800 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.801 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.801 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.801 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:11:17.801 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:17.133287+0000" 2024-09-22T03:11:17.801 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.801 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.801 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:11:17.801 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:17.086110+0000" 2024-09-22T03:11:17.801 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.801 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.801 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.801 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.801 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.801 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.802 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:11:17.802 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.802 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.802 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.802 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.802 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.802 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.802 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.802 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:11:17.802 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:17.088353+0000" 2024-09-22T03:11:17.802 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.802 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.802 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.803 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.803 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.803 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.803 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:11:17.803 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.803 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.803 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.803 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.803 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.803 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.803 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.803 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:11:17.803 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:17.088560+0000" 2024-09-22T03:11:17.803 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.804 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.804 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.804 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.804 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.804 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.804 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:11:17.804 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.804 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.804 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.804 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.804 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.804 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.804 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.804 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:11:17.804 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.805 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.805 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.805 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.805 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.805 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.805 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.805 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:11:17.805 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.805 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.805 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.805 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.805 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.805 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.805 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.806 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:11:17.806 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:17.088750+0000" 2024-09-22T03:11:17.806 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.806 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.806 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.806 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.806 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.806 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.806 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:11:17.806 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.806 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.806 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.806 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.807 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:17.088995+0000" 2024-09-22T03:11:17.807 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.807 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.807 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:11:17.807 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:17.097207+0000" 2024-09-22T03:11:17.807 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.807 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.807 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.807 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.807 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.807 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.807 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:11:17.807 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.807 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.808 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.808 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.808 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.808 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.808 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.808 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:11:17.808 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:17.132923+0000" 2024-09-22T03:11:17.808 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.808 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.808 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.808 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.808 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.808 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.808 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:11:17.809 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.809 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:11:17.809 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:11:17.809 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.809 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-22T03:11:17.809 INFO:teuthology.orchestra.run.smithi196.stdout: "type": "client_request", 2024-09-22T03:11:17.809 INFO:teuthology.orchestra.run.smithi196.stdout: "id": 479, 2024-09-22T03:11:17.809 INFO:teuthology.orchestra.run.smithi196.stdout: "detail": { 2024-09-22T03:11:17.809 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.809 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:11:17.809 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:17.090473+0000" 2024-09-22T03:11:17.809 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.809 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.809 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:11:17.810 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:17.090479+0000" 2024-09-22T03:11:17.810 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.810 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.810 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.810 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.810 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.810 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.810 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:11:17.810 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:17.090558+0000" 2024-09-22T03:11:17.810 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.810 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.810 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.810 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.810 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.811 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.811 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:11:17.811 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.811 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.811 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.811 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.811 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.811 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.811 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.811 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:11:17.811 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:17.090608+0000" 2024-09-22T03:11:17.811 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.811 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.811 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.812 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.812 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.812 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.812 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:11:17.812 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:17.091720+0000" 2024-09-22T03:11:17.812 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.812 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.812 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.812 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.812 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.812 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.812 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:11:17.812 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.812 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.812 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.813 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.813 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.813 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.813 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.813 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:11:17.813 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:17.139889+0000" 2024-09-22T03:11:17.813 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.813 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.813 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:11:17.813 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:17.092092+0000" 2024-09-22T03:11:17.813 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.813 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.813 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.813 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.814 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.814 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.814 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:11:17.814 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.814 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.814 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.814 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.814 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.814 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.814 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.814 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:11:17.814 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:17.092271+0000" 2024-09-22T03:11:17.814 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.815 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.815 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.815 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.815 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.815 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.815 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:11:17.815 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.815 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.815 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.815 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.815 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.815 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.815 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.815 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:11:17.816 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:17.092437+0000" 2024-09-22T03:11:17.816 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.816 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.816 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.816 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.816 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.816 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.816 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:11:17.816 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.816 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.816 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.816 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.816 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.816 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.816 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.817 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:11:17.817 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.817 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.817 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.817 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.817 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.817 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.817 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.817 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:11:17.817 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.817 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.817 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.817 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.817 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.818 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.818 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.818 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:11:17.818 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:17.092684+0000" 2024-09-22T03:11:17.818 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.818 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.818 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.818 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.818 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.818 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.818 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:11:17.818 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.818 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.818 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.819 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.819 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:17.093120+0000" 2024-09-22T03:11:17.819 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.819 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.819 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:11:17.819 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:17.100517+0000" 2024-09-22T03:11:17.819 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.819 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.819 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.819 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.819 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.819 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.819 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:11:17.819 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.820 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.820 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.820 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.820 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.820 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.820 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:11:17.820 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:11:17.820 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:11:17.139439+0000" 2024-09-22T03:11:17.820 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.820 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.820 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:11:17.820 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.820 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:11:17.820 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:11:17.820 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:11:17.821 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:11:17.821 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:11:17.821 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:11:17.821 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:11:17.821 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-22T03:11:17.821 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-09-22T03:11:18.266 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-22T03:11:18.304 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:2024-09-22T03:11:18.303991+0000 min lat: 0.00849108 max lat: 23.3792 avg lat: 0.155366 2024-09-22T03:11:18.304 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-22T03:11:18.304 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 80 16 8250 8234 0.804009 2.625 0.117235 0.155366 2024-09-22T03:11:18.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:11:18.404+0000 7fd0ec0d7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:11:18.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:11:18.411+0000 7fd0ec0d7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:11:19.304 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 81 16 8583 8567 0.826197 2.60156 0.0449854 0.151241 2024-09-22T03:11:20.305 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 82 16 8922 8906 0.848416 2.64844 0.0437717 0.147295 2024-09-22T03:11:20.633 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-22T03:11:20.771 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:11:20.768+0000 7f2ec5d57640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:11:20.775 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:11:20.774+0000 7f2ec5d57640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:11:21.305 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 83 16 9277 9261 0.871606 2.77344 0.0537972 0.143377 2024-09-22T03:11:22.305 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 84 16 9611 9595 0.89229 2.60938 0.0294435 0.140043 2024-09-22T03:11:22.823 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [2, 0, 1, 3] 2024-09-22T03:11:22.823 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-22T03:11:22.823 INFO:tasks.thrashosds.thrasher:Killing osd 3, live_osds are [2, 0, 1, 3] 2024-09-22T03:11:22.824 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-09-22T03:11:22.824 INFO:teuthology.orchestra.run:waiting for 300 2024-09-22T03:11:22.914 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-22T03:11:23.018 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:11:23.015+0000 7f9d01e18640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:11:23.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:11:23.019+0000 7f9d01e18640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:11:23.305 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 85 16 9803 9787 0.899438 1.5 0.018965 0.138185 2024-09-22T03:11:24.305 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 86 16 9803 9787 0.88898 0 - 0.138185 2024-09-22T03:11:25.305 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 87 16 9803 9787 0.878762 0 - 0.138185 2024-09-22T03:11:26.305 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 88 16 9803 9787 0.868775 0 - 0.138185 2024-09-22T03:11:26.861 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~0s 2024-09-22T03:11:27.305 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 89 16 9803 9787 0.859013 0 - 0.138185 2024-09-22T03:11:28.305 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 90 16 9803 9787 0.849469 0 - 0.138185 2024-09-22T03:11:28.924 INFO:tasks.ceph.osd.3:Stopped 2024-09-22T03:11:29.306 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 91 16 9803 9787 0.840134 0 - 0.138185 2024-09-22T03:11:30.306 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 92 16 9803 9787 0.831002 0 - 0.138185 2024-09-22T03:11:31.306 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 93 16 9803 9787 0.822066 0 - 0.138185 2024-09-22T03:11:32.306 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 94 16 9803 9787 0.813321 0 - 0.138185 2024-09-22T03:11:32.365 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 has been restored 2024-09-22T03:11:33.306 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 95 16 9803 9787 0.804759 0 - 0.138185 2024-09-22T03:11:33.925 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [3] live_osds: [2, 0, 1] 2024-09-22T03:11:33.925 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-22T03:11:33.925 INFO:tasks.thrashosds.thrasher:Reviving osd 3 2024-09-22T03:11:33.925 INFO:tasks.ceph.osd.3:Restarting daemon 2024-09-22T03:11:33.925 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2>> /var/log/ceph/ceph-osd.3.log 2024-09-22T03:11:33.928 INFO:tasks.ceph.osd.3:Started 2024-09-22T03:11:33.929 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-22T03:11:34.039 INFO:teuthology.orchestra.run.smithi196.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-09-22T03:11:34.042 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-22T03:11:34.043 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-09-22T03:11:34.306 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 96 16 9803 9787 0.796376 0 - 0.138185 2024-09-22T03:11:35.306 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 97 16 9803 9787 0.788166 0 - 0.138185 2024-09-22T03:11:36.307 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 98 16 9803 9787 0.780123 0 - 0.138185 2024-09-22T03:11:37.307 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 99 16 9803 9787 0.772242 0 - 0.138185 2024-09-22T03:11:38.307 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:2024-09-22T03:11:38.306496+0000 min lat: 0.00849108 max lat: 23.3792 avg lat: 0.138185 2024-09-22T03:11:38.307 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-22T03:11:38.307 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 100 16 9803 9787 0.76452 0 - 0.138185 2024-09-22T03:11:39.044 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-22T03:11:39.139 INFO:teuthology.orchestra.run.smithi196.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-09-22T03:11:39.142 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-22T03:11:39.142 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-09-22T03:11:39.307 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 101 16 9803 9787 0.75695 0 - 0.138185 2024-09-22T03:11:40.307 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 102 16 9803 9787 0.749528 0 - 0.138185 2024-09-22T03:11:41.220 INFO:teuthology.orchestra.run.smithi153.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-09-22T03:11:41.224 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-22T03:11:41.224 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-22T03:11:41.307 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 103 16 9803 9787 0.742252 0 - 0.138185 2024-09-22T03:11:42.307 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 104 16 9803 9787 0.735115 0 - 0.138185 2024-09-22T03:11:43.225 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-22T03:11:43.307 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 105 15 9803 9788 0.728188 0.000390625 20.5314 0.140269 2024-09-22T03:11:43.383 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:11:43.379+0000 7f9ccf397640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:11:43.386 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:11:43.384+0000 7f9ccf397640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:11:44.143 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-22T03:11:44.247 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-09-22T03:11:44.247 INFO:teuthology.orchestra.run.smithi196.stdout: "ops_in_flight": [] 2024-09-22T03:11:44.247 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-09-22T03:11:44.308 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:Total time run: 105.089 2024-09-22T03:11:44.308 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:Total writes made: 9803 2024-09-22T03:11:44.308 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:Write size: 8192 2024-09-22T03:11:44.308 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:Object size: 65536 2024-09-22T03:11:44.308 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:Bandwidth (MB/sec): 0.728769 2024-09-22T03:11:44.308 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:Stddev Bandwidth: 1.19838 2024-09-22T03:11:44.308 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:Max bandwidth (MB/sec): 3.125 2024-09-22T03:11:44.308 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:Min bandwidth (MB/sec): 0 2024-09-22T03:11:44.308 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:Average IOPS: 93 2024-09-22T03:11:44.308 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:Stddev IOPS: 153.399 2024-09-22T03:11:44.309 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:Max IOPS: 400 2024-09-22T03:11:44.309 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:Min IOPS: 0 2024-09-22T03:11:44.309 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:Average Latency(s): 0.171486 2024-09-22T03:11:44.309 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:Stddev Latency(s): 1.58433 2024-09-22T03:11:44.309 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:Max latency(s): 23.3792 2024-09-22T03:11:44.309 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:Min latency(s): 0.00849108 2024-09-22T03:11:44.321 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:Cleaning up (deleting benchmark objects) 2024-09-22T03:11:45.455 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-22T03:11:45.598 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:11:45.593+0000 7f17dbe58640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:11:45.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:11:45.601+0000 7f17dbe58640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:11:47.457 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:Removed 1226 objects 2024-09-22T03:11:47.457 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:Clean up completed and total clean up time :3.13607 2024-09-22T03:11:47.985 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-22T03:11:48.091 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:11:48.087+0000 7f55ef597640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:11:48.094 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:11:48.092+0000 7f55ef597640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:11:48.226 INFO:tasks.ceph.ceph_manager.ceph:removing pool_name unique_pool_1 2024-09-22T03:11:48.227 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm unique_pool_1 unique_pool_1 --yes-i-really-really-mean-it 2024-09-22T03:11:48.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:11:48.326+0000 7fd81f957640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:11:48.340 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:11:48.337+0000 7fd81f957640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:11:48.883 INFO:teuthology.orchestra.run.smithi153.stderr:pool 'unique_pool_1' removed 2024-09-22T03:11:48.901 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-09-22T03:11:48.902 INFO:tasks.radosbench:Beginning radosbench... 2024-09-22T03:11:48.902 INFO:tasks.ceph.ceph_manager.ceph:creating pool_name unique_pool_2 2024-09-22T03:11:48.902 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create unique_pool_2 16 2024-09-22T03:11:49.046 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:11:49.035+0000 7f851e917640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:11:49.050 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:11:49.047+0000 7f851e917640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:11:49.256 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [2, 0, 1, 3] 2024-09-22T03:11:49.256 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-22T03:11:49.256 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-09-22T03:11:49.256 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-09-22T03:11:49.361 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:11:49.355+0000 7f95faad7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:11:49.364 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:11:49.362+0000 7f95faad7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:11:49.377 INFO:teuthology.orchestra.run.smithi153.stderr:no valid command found; 10 closest matches: 2024-09-22T03:11:49.378 INFO:teuthology.orchestra.run.smithi153.stderr:config get 2024-09-22T03:11:49.378 INFO:teuthology.orchestra.run.smithi153.stderr:config help 2024-09-22T03:11:49.378 INFO:teuthology.orchestra.run.smithi153.stderr:config set ... 2024-09-22T03:11:49.378 INFO:teuthology.orchestra.run.smithi153.stderr:config show 2024-09-22T03:11:49.378 INFO:teuthology.orchestra.run.smithi153.stderr:deep_scrub 2024-09-22T03:11:49.378 INFO:teuthology.orchestra.run.smithi153.stderr:dump_historic_ops 2024-09-22T03:11:49.378 INFO:teuthology.orchestra.run.smithi153.stderr:dump_historic_slow_ops 2024-09-22T03:11:49.378 INFO:teuthology.orchestra.run.smithi153.stderr:dump_metrics [] 2024-09-22T03:11:49.378 INFO:teuthology.orchestra.run.smithi153.stderr:dump_ops_in_flight 2024-09-22T03:11:49.378 INFO:teuthology.orchestra.run.smithi153.stderr:dump_pgstate_history 2024-09-22T03:11:49.378 INFO:teuthology.orchestra.run.smithi153.stderr:Error EINVAL: invalid command 2024-09-22T03:11:49.380 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-22T03:11:49.380 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-22T03:11:49.381 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-09-22T03:11:49.486 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:11:49.480+0000 7fcffaf57640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:11:49.489 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:11:49.486+0000 7fcffaf57640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:11:49.502 INFO:teuthology.orchestra.run.smithi153.stderr:no valid command found; 10 closest matches: 2024-09-22T03:11:49.502 INFO:teuthology.orchestra.run.smithi153.stderr:config get 2024-09-22T03:11:49.502 INFO:teuthology.orchestra.run.smithi153.stderr:config help 2024-09-22T03:11:49.502 INFO:teuthology.orchestra.run.smithi153.stderr:config set ... 2024-09-22T03:11:49.502 INFO:teuthology.orchestra.run.smithi153.stderr:config show 2024-09-22T03:11:49.502 INFO:teuthology.orchestra.run.smithi153.stderr:deep_scrub 2024-09-22T03:11:49.502 INFO:teuthology.orchestra.run.smithi153.stderr:dump_historic_ops 2024-09-22T03:11:49.503 INFO:teuthology.orchestra.run.smithi153.stderr:dump_historic_slow_ops 2024-09-22T03:11:49.503 INFO:teuthology.orchestra.run.smithi153.stderr:dump_metrics [] 2024-09-22T03:11:49.503 INFO:teuthology.orchestra.run.smithi153.stderr:dump_ops_in_flight 2024-09-22T03:11:49.503 INFO:teuthology.orchestra.run.smithi153.stderr:dump_pgstate_history 2024-09-22T03:11:49.503 INFO:teuthology.orchestra.run.smithi153.stderr:Error EINVAL: invalid command 2024-09-22T03:11:49.505 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-22T03:11:49.505 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-22T03:11:49.505 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-09-22T03:11:49.611 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:11:49.605+0000 7fe84ac63640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:11:49.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:11:49.612+0000 7fe84ac63640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:11:49.627 INFO:teuthology.orchestra.run.smithi153.stderr:no valid command found; 10 closest matches: 2024-09-22T03:11:49.627 INFO:teuthology.orchestra.run.smithi153.stderr:config get 2024-09-22T03:11:49.627 INFO:teuthology.orchestra.run.smithi153.stderr:config help 2024-09-22T03:11:49.627 INFO:teuthology.orchestra.run.smithi153.stderr:config set ... 2024-09-22T03:11:49.627 INFO:teuthology.orchestra.run.smithi153.stderr:config show 2024-09-22T03:11:49.627 INFO:teuthology.orchestra.run.smithi153.stderr:deep_scrub 2024-09-22T03:11:49.627 INFO:teuthology.orchestra.run.smithi153.stderr:dump_historic_ops 2024-09-22T03:11:49.627 INFO:teuthology.orchestra.run.smithi153.stderr:dump_historic_slow_ops 2024-09-22T03:11:49.627 INFO:teuthology.orchestra.run.smithi153.stderr:dump_metrics [] 2024-09-22T03:11:49.627 INFO:teuthology.orchestra.run.smithi153.stderr:dump_ops_in_flight 2024-09-22T03:11:49.627 INFO:teuthology.orchestra.run.smithi153.stderr:dump_pgstate_history 2024-09-22T03:11:49.627 INFO:teuthology.orchestra.run.smithi153.stderr:Error EINVAL: invalid command 2024-09-22T03:11:49.629 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-22T03:11:49.630 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-22T03:11:49.630 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-09-22T03:11:49.735 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:11:49.729+0000 7f701a06c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:11:49.738 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:11:49.736+0000 7f701a06c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:11:49.750 INFO:teuthology.orchestra.run.smithi153.stderr:no valid command found; 10 closest matches: 2024-09-22T03:11:49.750 INFO:teuthology.orchestra.run.smithi153.stderr:config get 2024-09-22T03:11:49.750 INFO:teuthology.orchestra.run.smithi153.stderr:config help 2024-09-22T03:11:49.750 INFO:teuthology.orchestra.run.smithi153.stderr:config set ... 2024-09-22T03:11:49.751 INFO:teuthology.orchestra.run.smithi153.stderr:config show 2024-09-22T03:11:49.751 INFO:teuthology.orchestra.run.smithi153.stderr:deep_scrub 2024-09-22T03:11:49.751 INFO:teuthology.orchestra.run.smithi153.stderr:dump_historic_ops 2024-09-22T03:11:49.751 INFO:teuthology.orchestra.run.smithi153.stderr:dump_historic_slow_ops 2024-09-22T03:11:49.751 INFO:teuthology.orchestra.run.smithi153.stderr:dump_metrics [] 2024-09-22T03:11:49.751 INFO:teuthology.orchestra.run.smithi153.stderr:dump_ops_in_flight 2024-09-22T03:11:49.751 INFO:teuthology.orchestra.run.smithi153.stderr:dump_pgstate_history 2024-09-22T03:11:49.751 INFO:teuthology.orchestra.run.smithi153.stderr:Error EINVAL: invalid command 2024-09-22T03:11:49.753 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-22T03:11:49.754 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-22T03:11:49.887 INFO:teuthology.orchestra.run.smithi153.stderr:pool 'unique_pool_2' created 2024-09-22T03:11:49.903 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool application enable unique_pool_2 rados --yes-i-really-mean-it || true 2024-09-22T03:11:50.022 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:11:50.011+0000 7febe7397640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:11:50.025 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:11:50.023+0000 7febe7397640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:11:50.164 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-22T03:11:50.266 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:11:50.263+0000 7fdfa4e2c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:11:50.269 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:11:50.268+0000 7fdfa4e2c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:11:50.890 INFO:teuthology.orchestra.run.smithi153.stderr:enabled application 'rados' on pool 'unique_pool_2' 2024-09-22T03:11:51.910 DEBUG:teuthology.orchestra.run.smithi153:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 8192 --object-size 65536 -p unique_pool_2 bench 90 write' 2024-09-22T03:11:51.913 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-09-22T03:11:51.913 INFO:teuthology.orchestra.run:waiting for 3000 2024-09-22T03:11:51.971 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:hints = 1 2024-09-22T03:11:51.971 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:Maintaining 16 concurrent writes of 8192 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-09-22T03:11:51.971 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:Object prefix: benchmark_data_smithi153_33956 2024-09-22T03:11:51.971 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-22T03:11:51.971 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 0 0 0 0 0 0 - 0 2024-09-22T03:11:52.338 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-22T03:11:52.478 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:11:52.474+0000 7f112484f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:11:52.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:11:52.480+0000 7f112484f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:11:52.971 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 1 16 375 359 2.80452 2.80469 0.0250758 0.0433939 2024-09-22T03:11:53.971 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 2 16 780 764 2.98417 3.16406 0.029146 0.0413591 2024-09-22T03:11:54.585 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-22T03:11:54.727 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:11:54.723+0000 7f478ff97640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:11:54.732 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:11:54.730+0000 7f478ff97640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:11:54.755 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [2, 0, 1, 3] 2024-09-22T03:11:54.755 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-22T03:11:54.755 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-09-22T03:11:54.756 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-09-22T03:11:54.893 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:11:54.882+0000 7f5bbf197640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:11:54.900 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:11:54.898+0000 7f5bbf197640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:11:54.971 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 3 16 1138 1122 2.92167 2.79688 0.0139061 0.0424483 2024-09-22T03:11:54.976 INFO:teuthology.orchestra.run.smithi153.stderr:no valid command found; 10 closest matches: 2024-09-22T03:11:54.976 INFO:teuthology.orchestra.run.smithi153.stderr:config get 2024-09-22T03:11:54.976 INFO:teuthology.orchestra.run.smithi153.stderr:config help 2024-09-22T03:11:54.976 INFO:teuthology.orchestra.run.smithi153.stderr:config set ... 2024-09-22T03:11:54.976 INFO:teuthology.orchestra.run.smithi153.stderr:config show 2024-09-22T03:11:54.976 INFO:teuthology.orchestra.run.smithi153.stderr:deep_scrub 2024-09-22T03:11:54.976 INFO:teuthology.orchestra.run.smithi153.stderr:dump_historic_ops 2024-09-22T03:11:54.976 INFO:teuthology.orchestra.run.smithi153.stderr:dump_historic_slow_ops 2024-09-22T03:11:54.976 INFO:teuthology.orchestra.run.smithi153.stderr:dump_metrics [] 2024-09-22T03:11:54.976 INFO:teuthology.orchestra.run.smithi153.stderr:dump_ops_in_flight 2024-09-22T03:11:54.977 INFO:teuthology.orchestra.run.smithi153.stderr:dump_pgstate_history 2024-09-22T03:11:54.977 INFO:teuthology.orchestra.run.smithi153.stderr:Error EINVAL: invalid command 2024-09-22T03:11:54.980 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-22T03:11:54.980 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-22T03:11:54.981 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-09-22T03:11:55.158 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:11:55.150+0000 7f5612ab2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:11:55.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:11:55.162+0000 7f5612ab2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:11:55.218 INFO:teuthology.orchestra.run.smithi153.stderr:no valid command found; 10 closest matches: 2024-09-22T03:11:55.218 INFO:teuthology.orchestra.run.smithi153.stderr:config get 2024-09-22T03:11:55.219 INFO:teuthology.orchestra.run.smithi153.stderr:config help 2024-09-22T03:11:55.219 INFO:teuthology.orchestra.run.smithi153.stderr:config set ... 2024-09-22T03:11:55.219 INFO:teuthology.orchestra.run.smithi153.stderr:config show 2024-09-22T03:11:55.219 INFO:teuthology.orchestra.run.smithi153.stderr:deep_scrub 2024-09-22T03:11:55.219 INFO:teuthology.orchestra.run.smithi153.stderr:dump_historic_ops 2024-09-22T03:11:55.219 INFO:teuthology.orchestra.run.smithi153.stderr:dump_historic_slow_ops 2024-09-22T03:11:55.219 INFO:teuthology.orchestra.run.smithi153.stderr:dump_metrics [] 2024-09-22T03:11:55.219 INFO:teuthology.orchestra.run.smithi153.stderr:dump_ops_in_flight 2024-09-22T03:11:55.219 INFO:teuthology.orchestra.run.smithi153.stderr:dump_pgstate_history 2024-09-22T03:11:55.219 INFO:teuthology.orchestra.run.smithi153.stderr:Error EINVAL: invalid command 2024-09-22T03:11:55.221 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-22T03:11:55.221 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-22T03:11:55.222 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-09-22T03:11:55.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:11:55.357+0000 7f331b797640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:11:55.371 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:11:55.370+0000 7f331b797640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:11:55.407 INFO:teuthology.orchestra.run.smithi153.stderr:no valid command found; 10 closest matches: 2024-09-22T03:11:55.407 INFO:teuthology.orchestra.run.smithi153.stderr:config get 2024-09-22T03:11:55.407 INFO:teuthology.orchestra.run.smithi153.stderr:config help 2024-09-22T03:11:55.407 INFO:teuthology.orchestra.run.smithi153.stderr:config set ... 2024-09-22T03:11:55.407 INFO:teuthology.orchestra.run.smithi153.stderr:config show 2024-09-22T03:11:55.408 INFO:teuthology.orchestra.run.smithi153.stderr:deep_scrub 2024-09-22T03:11:55.408 INFO:teuthology.orchestra.run.smithi153.stderr:dump_historic_ops 2024-09-22T03:11:55.408 INFO:teuthology.orchestra.run.smithi153.stderr:dump_historic_slow_ops 2024-09-22T03:11:55.408 INFO:teuthology.orchestra.run.smithi153.stderr:dump_metrics [] 2024-09-22T03:11:55.408 INFO:teuthology.orchestra.run.smithi153.stderr:dump_ops_in_flight 2024-09-22T03:11:55.408 INFO:teuthology.orchestra.run.smithi153.stderr:dump_pgstate_history 2024-09-22T03:11:55.408 INFO:teuthology.orchestra.run.smithi153.stderr:Error EINVAL: invalid command 2024-09-22T03:11:55.411 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-22T03:11:55.411 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-22T03:11:55.411 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-09-22T03:11:55.567 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:11:55.556+0000 7f43ac30e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:11:55.571 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:11:55.569+0000 7f43ac30e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:11:55.588 INFO:teuthology.orchestra.run.smithi153.stderr:no valid command found; 10 closest matches: 2024-09-22T03:11:55.588 INFO:teuthology.orchestra.run.smithi153.stderr:config get 2024-09-22T03:11:55.588 INFO:teuthology.orchestra.run.smithi153.stderr:config help 2024-09-22T03:11:55.589 INFO:teuthology.orchestra.run.smithi153.stderr:config set ... 2024-09-22T03:11:55.589 INFO:teuthology.orchestra.run.smithi153.stderr:config show 2024-09-22T03:11:55.589 INFO:teuthology.orchestra.run.smithi153.stderr:deep_scrub 2024-09-22T03:11:55.589 INFO:teuthology.orchestra.run.smithi153.stderr:dump_historic_ops 2024-09-22T03:11:55.589 INFO:teuthology.orchestra.run.smithi153.stderr:dump_historic_slow_ops 2024-09-22T03:11:55.589 INFO:teuthology.orchestra.run.smithi153.stderr:dump_metrics [] 2024-09-22T03:11:55.589 INFO:teuthology.orchestra.run.smithi153.stderr:dump_ops_in_flight 2024-09-22T03:11:55.589 INFO:teuthology.orchestra.run.smithi153.stderr:dump_pgstate_history 2024-09-22T03:11:55.589 INFO:teuthology.orchestra.run.smithi153.stderr:Error EINVAL: invalid command 2024-09-22T03:11:55.592 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-22T03:11:55.592 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-22T03:11:55.971 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 4 16 1526 1510 2.949 3.03125 0.0360122 0.042152 2024-09-22T03:11:56.872 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-22T03:11:56.971 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 5 16 1903 1887 2.94821 2.94531 0.0663575 0.0421125 2024-09-22T03:11:57.022 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:11:57.018+0000 7f49bae6c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:11:57.027 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:11:57.026+0000 7f49bae6c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:11:57.971 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 6 16 2282 2266 2.95029 2.96094 0.0479201 0.0421351 2024-09-22T03:11:58.971 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 7 16 2653 2637 2.94285 2.89844 0.028578 0.0423147 2024-09-22T03:11:59.237 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-22T03:11:59.371 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:11:59.368+0000 7f522942c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:11:59.375 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:11:59.373+0000 7f522942c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:11:59.971 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 8 16 3005 2989 2.91871 2.75 0.0529371 0.0426862 2024-09-22T03:12:00.593 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [2, 0, 1, 3] 2024-09-22T03:12:00.593 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-22T03:12:00.593 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-22T03:12:00.745 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:12:00.739+0000 7fcf982d7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:12:00.750 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:12:00.748+0000 7fcf982d7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:12:00.971 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 9 16 3379 3363 2.91905 2.92188 0.0371395 0.0426856 2024-09-22T03:12:00.988 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-22T03:12:00.989 INFO:teuthology.orchestra.run.smithi153.stderr:dumped all 2024-09-22T03:12:01.008 INFO:teuthology.orchestra.run.smithi153.stdout:{"pg_ready":true,"pg_map":{"version":163,"stamp":"2024-09-22T03:12:00.154424+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19464755,"num_objects":292,"num_object_clones":0,"num_object_copies":876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":2443,"num_write_kb":20328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2388,"ondisk_log_size":2388,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":163436,"kb_used_data":24620,"kb_used_omap":0,"kb_used_meta":138752,"kb_avail":377323924,"statfs":{"total":386547056640,"available":386379698176,"internally_reserved":0,"allocated":25210880,"data_stored":49012753,"data_compressed":2492157,"data_compressed_allocated":23416832,"data_compressed_original":48308224,"omap_allocated":0,"internal_metadata":142082048},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":18874368,"num_objects":288,"num_object_clones":0,"num_object_copies":864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2304,"num_write_kb":18432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.264222"},"pg_stats":[{"pgid":"5.f","version":"51'79","reported_seq":92,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-22T03:11:56.661265+0000","last_change":"2024-09-22T03:11:50.987253+0000","last_active":"2024-09-22T03:11:56.661265+0000","last_peered":"2024-09-22T03:11:56.661265+0000","last_clean":"2024-09-22T03:11:56.661265+0000","last_became_active":"2024-09-22T03:11:50.981486+0000","last_became_peered":"2024-09-22T03:11:50.981486+0000","last_unstale":"2024-09-22T03:11:56.661265+0000","last_undegraded":"2024-09-22T03:11:56.661265+0000","last_fullsized":"2024-09-22T03:11:56.661265+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:11:49.882601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:11:49.882601+0000","last_clean_scrub_stamp":"2024-09-22T03:11:49.882601+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":655360,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":80,"num_write_kb":640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.e","version":"51'175","reported_seq":188,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-22T03:11:57.932324+0000","last_change":"2024-09-22T03:11:50.991544+0000","last_active":"2024-09-22T03:11:57.932324+0000","last_peered":"2024-09-22T03:11:57.932324+0000","last_clean":"2024-09-22T03:11:57.932324+0000","last_became_active":"2024-09-22T03:11:50.987680+0000","last_became_peered":"2024-09-22T03:11:50.987680+0000","last_unstale":"2024-09-22T03:11:57.932324+0000","last_undegraded":"2024-09-22T03:11:57.932324+0000","last_fullsized":"2024-09-22T03:11:57.932324+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:11:49.882601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:11:49.882601+0000","last_clean_scrub_stamp":"2024-09-22T03:11:49.882601+0000","objects_scrubbed":0,"log_size":175,"log_dups_size":0,"ondisk_log_size":175,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1441792,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":176,"num_write_kb":1408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.d","version":"51'247","reported_seq":260,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-22T03:11:59.696807+0000","last_change":"2024-09-22T03:11:50.983002+0000","last_active":"2024-09-22T03:11:59.696807+0000","last_peered":"2024-09-22T03:11:59.696807+0000","last_clean":"2024-09-22T03:11:59.696807+0000","last_became_active":"2024-09-22T03:11:50.978025+0000","last_became_peered":"2024-09-22T03:11:50.978025+0000","last_unstale":"2024-09-22T03:11:59.696807+0000","last_undegraded":"2024-09-22T03:11:59.696807+0000","last_fullsized":"2024-09-22T03:11:59.696807+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:11:49.882601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:11:49.882601+0000","last_clean_scrub_stamp":"2024-09-22T03:11:49.882601+0000","objects_scrubbed":0,"log_size":247,"log_dups_size":0,"ondisk_log_size":247,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2031616,"num_objects":31,"num_object_clones":0,"num_object_copies":93,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":248,"num_write_kb":1984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.c","version":"51'55","reported_seq":68,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-22T03:11:55.109234+0000","last_change":"2024-09-22T03:11:50.984233+0000","last_active":"2024-09-22T03:11:55.109234+0000","last_peered":"2024-09-22T03:11:55.109234+0000","last_clean":"2024-09-22T03:11:55.109234+0000","last_became_active":"2024-09-22T03:11:50.979546+0000","last_became_peered":"2024-09-22T03:11:50.979546+0000","last_unstale":"2024-09-22T03:11:55.109234+0000","last_undegraded":"2024-09-22T03:11:55.109234+0000","last_fullsized":"2024-09-22T03:11:55.109234+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:11:49.882601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:11:49.882601+0000","last_clean_scrub_stamp":"2024-09-22T03:11:49.882601+0000","objects_scrubbed":0,"log_size":55,"log_dups_size":0,"ondisk_log_size":55,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":458752,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":56,"num_write_kb":448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.b","version":"51'191","reported_seq":204,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-22T03:11:59.670094+0000","last_change":"2024-09-22T03:11:50.953185+0000","last_active":"2024-09-22T03:11:59.670094+0000","last_peered":"2024-09-22T03:11:59.670094+0000","last_clean":"2024-09-22T03:11:59.670094+0000","last_became_active":"2024-09-22T03:11:50.948721+0000","last_became_peered":"2024-09-22T03:11:50.948721+0000","last_unstale":"2024-09-22T03:11:59.670094+0000","last_undegraded":"2024-09-22T03:11:59.670094+0000","last_fullsized":"2024-09-22T03:11:59.670094+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:11:49.882601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:11:49.882601+0000","last_clean_scrub_stamp":"2024-09-22T03:11:49.882601+0000","objects_scrubbed":0,"log_size":191,"log_dups_size":0,"ondisk_log_size":191,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1572864,"num_objects":24,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":192,"num_write_kb":1536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"51'167","reported_seq":180,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-22T03:11:57.122741+0000","last_change":"2024-09-22T03:11:50.968686+0000","last_active":"2024-09-22T03:11:57.122741+0000","last_peered":"2024-09-22T03:11:57.122741+0000","last_clean":"2024-09-22T03:11:57.122741+0000","last_became_active":"2024-09-22T03:11:50.957056+0000","last_became_peered":"2024-09-22T03:11:50.957056+0000","last_unstale":"2024-09-22T03:11:57.122741+0000","last_undegraded":"2024-09-22T03:11:57.122741+0000","last_fullsized":"2024-09-22T03:11:57.122741+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:11:49.882601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:11:49.882601+0000","last_clean_scrub_stamp":"2024-09-22T03:11:49.882601+0000","objects_scrubbed":0,"log_size":167,"log_dups_size":0,"ondisk_log_size":167,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1376256,"num_objects":21,"num_object_clones":0,"num_object_copies":63,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":168,"num_write_kb":1344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.9","version":"51'127","reported_seq":140,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-22T03:11:55.971645+0000","last_change":"2024-09-22T03:11:50.952158+0000","last_active":"2024-09-22T03:11:55.971645+0000","last_peered":"2024-09-22T03:11:55.971645+0000","last_clean":"2024-09-22T03:11:55.971645+0000","last_became_active":"2024-09-22T03:11:50.947858+0000","last_became_peered":"2024-09-22T03:11:50.947858+0000","last_unstale":"2024-09-22T03:11:55.971645+0000","last_undegraded":"2024-09-22T03:11:55.971645+0000","last_fullsized":"2024-09-22T03:11:55.971645+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:11:49.882601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:11:49.882601+0000","last_clean_scrub_stamp":"2024-09-22T03:11:49.882601+0000","objects_scrubbed":0,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1048576,"num_objects":16,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":128,"num_write_kb":1024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.8","version":"51'199","reported_seq":212,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-22T03:11:59.495521+0000","last_change":"2024-09-22T03:11:50.971035+0000","last_active":"2024-09-22T03:11:59.495521+0000","last_peered":"2024-09-22T03:11:59.495521+0000","last_clean":"2024-09-22T03:11:59.495521+0000","last_became_active":"2024-09-22T03:11:50.966928+0000","last_became_peered":"2024-09-22T03:11:50.966928+0000","last_unstale":"2024-09-22T03:11:59.495521+0000","last_undegraded":"2024-09-22T03:11:59.495521+0000","last_fullsized":"2024-09-22T03:11:59.495521+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:11:49.882601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:11:49.882601+0000","last_clean_scrub_stamp":"2024-09-22T03:11:49.882601+0000","objects_scrubbed":0,"log_size":199,"log_dups_size":0,"ondisk_log_size":199,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1638400,"num_objects":25,"num_object_clones":0,"num_object_copies":75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":200,"num_write_kb":1600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.4","version":"51'103","reported_seq":116,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-22T03:11:56.890333+0000","last_change":"2024-09-22T03:11:50.981892+0000","last_active":"2024-09-22T03:11:56.890333+0000","last_peered":"2024-09-22T03:11:56.890333+0000","last_clean":"2024-09-22T03:11:56.890333+0000","last_became_active":"2024-09-22T03:11:50.978317+0000","last_became_peered":"2024-09-22T03:11:50.978317+0000","last_unstale":"2024-09-22T03:11:56.890333+0000","last_undegraded":"2024-09-22T03:11:56.890333+0000","last_fullsized":"2024-09-22T03:11:56.890333+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:11:49.882601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:11:49.882601+0000","last_clean_scrub_stamp":"2024-09-22T03:11:49.882601+0000","objects_scrubbed":0,"log_size":103,"log_dups_size":0,"ondisk_log_size":103,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":851968,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":104,"num_write_kb":832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":56,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-22T03:11:51.897529+0000","last_change":"2024-09-22T03:11:48.924496+0000","last_active":"2024-09-22T03:11:51.897529+0000","last_peered":"2024-09-22T03:11:51.897529+0000","last_clean":"2024-09-22T03:11:51.897529+0000","last_became_active":"2024-09-22T03:11:43.308297+0000","last_became_peered":"2024-09-22T03:11:43.308297+0000","last_unstale":"2024-09-22T03:11:51.897529+0000","last_undegraded":"2024-09-22T03:11:51.897529+0000","last_fullsized":"2024-09-22T03:11:51.897529+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"51'111","reported_seq":124,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-22T03:11:57.013455+0000","last_change":"2024-09-22T03:11:50.971039+0000","last_active":"2024-09-22T03:11:57.013455+0000","last_peered":"2024-09-22T03:11:57.013455+0000","last_clean":"2024-09-22T03:11:57.013455+0000","last_became_active":"2024-09-22T03:11:50.965225+0000","last_became_peered":"2024-09-22T03:11:50.965225+0000","last_unstale":"2024-09-22T03:11:57.013455+0000","last_undegraded":"2024-09-22T03:11:57.013455+0000","last_fullsized":"2024-09-22T03:11:57.013455+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:11:49.882601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:11:49.882601+0000","last_clean_scrub_stamp":"2024-09-22T03:11:49.882601+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":917504,"num_objects":14,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":112,"num_write_kb":896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":119,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-22T03:11:51.903177+0000","last_change":"2024-09-22T03:11:13.920007+0000","last_active":"2024-09-22T03:11:51.903177+0000","last_peered":"2024-09-22T03:11:51.903177+0000","last_clean":"2024-09-22T03:11:51.903177+0000","last_became_active":"2024-09-22T03:10:52.971952+0000","last_became_peered":"2024-09-22T03:10:52.971952+0000","last_unstale":"2024-09-22T03:11:51.903177+0000","last_undegraded":"2024-09-22T03:11:51.903177+0000","last_fullsized":"2024-09-22T03:11:51.903177+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.6","version":"51'119","reported_seq":132,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-22T03:11:56.978070+0000","last_change":"2024-09-22T03:11:50.984462+0000","last_active":"2024-09-22T03:11:56.978070+0000","last_peered":"2024-09-22T03:11:56.978070+0000","last_clean":"2024-09-22T03:11:56.978070+0000","last_became_active":"2024-09-22T03:11:50.978017+0000","last_became_peered":"2024-09-22T03:11:50.978017+0000","last_unstale":"2024-09-22T03:11:56.978070+0000","last_undegraded":"2024-09-22T03:11:56.978070+0000","last_fullsized":"2024-09-22T03:11:56.978070+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:11:49.882601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:11:49.882601+0000","last_clean_scrub_stamp":"2024-09-22T03:11:49.882601+0000","objects_scrubbed":0,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":983040,"num_objects":15,"num_object_clones":0,"num_object_copies":45,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":121,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-22T03:11:51.901364+0000","last_change":"2024-09-22T03:11:48.924593+0000","last_active":"2024-09-22T03:11:51.901364+0000","last_peered":"2024-09-22T03:11:51.901364+0000","last_clean":"2024-09-22T03:11:51.901364+0000","last_became_active":"2024-09-22T03:11:43.313592+0000","last_became_peered":"2024-09-22T03:11:43.313592+0000","last_unstale":"2024-09-22T03:11:51.901364+0000","last_undegraded":"2024-09-22T03:11:51.901364+0000","last_fullsized":"2024-09-22T03:11:51.901364+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.7","version":"51'199","reported_seq":212,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-22T03:11:59.603432+0000","last_change":"2024-09-22T03:11:50.980497+0000","last_active":"2024-09-22T03:11:59.603432+0000","last_peered":"2024-09-22T03:11:59.603432+0000","last_clean":"2024-09-22T03:11:59.603432+0000","last_became_active":"2024-09-22T03:11:50.975466+0000","last_became_peered":"2024-09-22T03:11:50.975466+0000","last_unstale":"2024-09-22T03:11:59.603432+0000","last_undegraded":"2024-09-22T03:11:59.603432+0000","last_fullsized":"2024-09-22T03:11:59.603432+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:11:49.882601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:11:49.882601+0000","last_clean_scrub_stamp":"2024-09-22T03:11:49.882601+0000","objects_scrubbed":0,"log_size":199,"log_dups_size":0,"ondisk_log_size":199,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1638400,"num_objects":25,"num_object_clones":0,"num_object_copies":75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":200,"num_write_kb":1600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":99,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-22T03:11:51.896242+0000","last_change":"2024-09-22T03:11:48.926622+0000","last_active":"2024-09-22T03:11:51.896242+0000","last_peered":"2024-09-22T03:11:51.896242+0000","last_clean":"2024-09-22T03:11:51.896242+0000","last_became_active":"2024-09-22T03:11:43.322841+0000","last_became_peered":"2024-09-22T03:11:43.322841+0000","last_unstale":"2024-09-22T03:11:51.896242+0000","last_undegraded":"2024-09-22T03:11:51.896242+0000","last_fullsized":"2024-09-22T03:11:51.896242+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"12'97","reported_seq":243,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-22T03:11:51.897639+0000","last_change":"2024-09-22T03:10:52.959989+0000","last_active":"2024-09-22T03:11:51.897639+0000","last_peered":"2024-09-22T03:11:51.897639+0000","last_clean":"2024-09-22T03:11:51.897639+0000","last_became_active":"2024-09-22T03:10:52.950059+0000","last_became_peered":"2024-09-22T03:10:52.950059+0000","last_unstale":"2024-09-22T03:11:51.897639+0000","last_undegraded":"2024-09-22T03:11:51.897639+0000","last_fullsized":"2024-09-22T03:11:51.897639+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"51'175","reported_seq":188,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-22T03:11:59.191263+0000","last_change":"2024-09-22T03:11:51.001600+0000","last_active":"2024-09-22T03:11:59.191263+0000","last_peered":"2024-09-22T03:11:59.191263+0000","last_clean":"2024-09-22T03:11:59.191263+0000","last_became_active":"2024-09-22T03:11:50.995429+0000","last_became_peered":"2024-09-22T03:11:50.995429+0000","last_unstale":"2024-09-22T03:11:59.191263+0000","last_undegraded":"2024-09-22T03:11:59.191263+0000","last_fullsized":"2024-09-22T03:11:59.191263+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:11:49.882601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:11:49.882601+0000","last_clean_scrub_stamp":"2024-09-22T03:11:49.882601+0000","objects_scrubbed":0,"log_size":175,"log_dups_size":0,"ondisk_log_size":175,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1441792,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":176,"num_write_kb":1408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":111,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-22T03:11:51.900308+0000","last_change":"2024-09-22T03:11:13.920107+0000","last_active":"2024-09-22T03:11:51.900308+0000","last_peered":"2024-09-22T03:11:51.900308+0000","last_clean":"2024-09-22T03:11:51.900308+0000","last_became_active":"2024-09-22T03:10:52.991725+0000","last_became_peered":"2024-09-22T03:10:52.991725+0000","last_unstale":"2024-09-22T03:11:51.900308+0000","last_undegraded":"2024-09-22T03:11:51.900308+0000","last_fullsized":"2024-09-22T03:11:51.900308+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1","version":"51'87","reported_seq":100,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-22T03:11:56.158466+0000","last_change":"2024-09-22T03:11:50.970001+0000","last_active":"2024-09-22T03:11:56.158466+0000","last_peered":"2024-09-22T03:11:56.158466+0000","last_clean":"2024-09-22T03:11:56.158466+0000","last_became_active":"2024-09-22T03:11:50.963755+0000","last_became_peered":"2024-09-22T03:11:50.963755+0000","last_unstale":"2024-09-22T03:11:56.158466+0000","last_undegraded":"2024-09-22T03:11:56.158466+0000","last_fullsized":"2024-09-22T03:11:56.158466+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:11:49.882601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:11:49.882601+0000","last_clean_scrub_stamp":"2024-09-22T03:11:49.882601+0000","objects_scrubbed":0,"log_size":87,"log_dups_size":0,"ondisk_log_size":87,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":720896,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":88,"num_write_kb":704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":108,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-22T03:11:51.897554+0000","last_change":"2024-09-22T03:11:48.912413+0000","last_active":"2024-09-22T03:11:51.897554+0000","last_peered":"2024-09-22T03:11:51.897554+0000","last_clean":"2024-09-22T03:11:51.897554+0000","last_became_active":"2024-09-22T03:11:43.308745+0000","last_became_peered":"2024-09-22T03:11:43.308745+0000","last_unstale":"2024-09-22T03:11:51.897554+0000","last_undegraded":"2024-09-22T03:11:51.897554+0000","last_fullsized":"2024-09-22T03:11:51.897554+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2","version":"51'175","reported_seq":188,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-22T03:11:59.726986+0000","last_change":"2024-09-22T03:11:50.980354+0000","last_active":"2024-09-22T03:11:59.726986+0000","last_peered":"2024-09-22T03:11:59.726986+0000","last_clean":"2024-09-22T03:11:59.726986+0000","last_became_active":"2024-09-22T03:11:50.975134+0000","last_became_peered":"2024-09-22T03:11:50.975134+0000","last_unstale":"2024-09-22T03:11:59.726986+0000","last_undegraded":"2024-09-22T03:11:59.726986+0000","last_fullsized":"2024-09-22T03:11:59.726986+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:11:49.882601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:11:49.882601+0000","last_clean_scrub_stamp":"2024-09-22T03:11:49.882601+0000","objects_scrubbed":0,"log_size":175,"log_dups_size":0,"ondisk_log_size":175,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1441792,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":176,"num_write_kb":1408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":99,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-22T03:11:51.901559+0000","last_change":"2024-09-22T03:11:48.926233+0000","last_active":"2024-09-22T03:11:51.901559+0000","last_peered":"2024-09-22T03:11:51.901559+0000","last_clean":"2024-09-22T03:11:51.901559+0000","last_became_active":"2024-09-22T03:11:43.336676+0000","last_became_peered":"2024-09-22T03:11:43.336676+0000","last_unstale":"2024-09-22T03:11:51.901559+0000","last_undegraded":"2024-09-22T03:11:51.901559+0000","last_fullsized":"2024-09-22T03:11:51.901559+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"51'79","reported_seq":92,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-22T03:11:56.641605+0000","last_change":"2024-09-22T03:11:50.960247+0000","last_active":"2024-09-22T03:11:56.641605+0000","last_peered":"2024-09-22T03:11:56.641605+0000","last_clean":"2024-09-22T03:11:56.641605+0000","last_became_active":"2024-09-22T03:11:50.947625+0000","last_became_peered":"2024-09-22T03:11:50.947625+0000","last_unstale":"2024-09-22T03:11:56.641605+0000","last_undegraded":"2024-09-22T03:11:56.641605+0000","last_fullsized":"2024-09-22T03:11:56.641605+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:11:49.882601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:11:49.882601+0000","last_clean_scrub_stamp":"2024-09-22T03:11:49.882601+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":655360,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":80,"num_write_kb":640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":109,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-22T03:11:51.899344+0000","last_change":"2024-09-22T03:11:48.912598+0000","last_active":"2024-09-22T03:11:51.899344+0000","last_peered":"2024-09-22T03:11:51.899344+0000","last_clean":"2024-09-22T03:11:51.899344+0000","last_became_active":"2024-09-22T03:11:43.297297+0000","last_became_peered":"2024-09-22T03:11:43.297297+0000","last_unstale":"2024-09-22T03:11:51.899344+0000","last_undegraded":"2024-09-22T03:11:51.899344+0000","last_fullsized":"2024-09-22T03:11:51.899344+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":16,"stat_sum":{"num_bytes":18874368,"num_objects":288,"num_object_clones":0,"num_object_copies":864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2304,"num_write_kb":18432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":25333760,"data_stored":50667520,"data_compressed":2612925,"data_compressed_allocated":25333760,"data_compressed_original":50667520,"omap_allocated":0,"internal_metadata":0},"log_size":2288,"ondisk_log_size":2288,"up":48,"acting":48,"num_store_stats":4},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1180736,"data_compressed":59924,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":46,"seq":197568495629,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31620,"kb_used_data":3572,"kb_used_omap":0,"kb_used_meta":28032,"kb_avail":94340220,"statfs":{"total":96636764160,"available":96604385280,"internally_reserved":0,"allocated":3657728,"data_stored":6610545,"data_compressed":332157,"data_compressed_allocated":3223552,"data_compressed_original":6447104,"omap_allocated":0,"internal_metadata":28704768},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":19,"seq":81604378818,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":42644,"kb_used_data":7236,"kb_used_omap":0,"kb_used_meta":35392,"kb_avail":94329196,"statfs":{"total":96636764160,"available":96593096704,"internally_reserved":0,"allocated":7409664,"data_stored":14581905,"data_compressed":742980,"data_compressed_allocated":6955008,"data_compressed_original":14401536,"omap_allocated":0,"internal_metadata":36241408},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":40,"seq":171798691904,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":43936,"kb_used_data":3984,"kb_used_omap":0,"kb_used_meta":39936,"kb_avail":94327904,"statfs":{"total":96636764160,"available":96591773696,"internally_reserved":0,"allocated":4079616,"data_stored":7921809,"data_compressed":399433,"data_compressed_allocated":3624960,"data_compressed_original":7741440,"omap_allocated":0,"internal_metadata":40894464},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":27,"seq":115964117122,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":45236,"kb_used_data":9828,"kb_used_omap":0,"kb_used_meta":35392,"kb_avail":94326604,"statfs":{"total":96636764160,"available":96590442496,"internally_reserved":0,"allocated":10063872,"data_stored":19898494,"data_compressed":1017587,"data_compressed_allocated":9613312,"data_compressed_original":19718144,"omap_allocated":0,"internal_metadata":36241408},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29962,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29962,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":10117120,"data_stored":20234240,"data_compressed":1043884,"data_compressed_allocated":10117120,"data_compressed_original":20234240,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3710976,"data_stored":7421952,"data_compressed":382584,"data_compressed_allocated":3710976,"data_compressed_original":7421952,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7147520,"data_stored":14295040,"data_compressed":737129,"data_compressed_allocated":7147520,"data_compressed_original":14295040,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4358144,"data_stored":8716288,"data_compressed":449328,"data_compressed_allocated":4358144,"data_compressed_original":8716288,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-22T03:12:01.532 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-22T03:12:01.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:12:01.668+0000 7f933f74e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:12:01.678 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:12:01.676+0000 7f933f74e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:12:01.971 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 10 16 3773 3757 2.93493 3.07812 0.0375849 0.0425218 2024-09-22T03:12:02.971 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 11 16 4136 4120 2.92591 2.83594 0.0384697 0.0425866 2024-09-22T03:12:03.910 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-22T03:12:03.971 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 12 16 4496 4480 2.91644 2.8125 0.0546071 0.0428101 2024-09-22T03:12:04.029 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:12:04.026+0000 7fc6cdb97640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:12:04.033 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:12:04.032+0000 7fc6cdb97640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:12:04.972 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 13 16 4835 4819 2.89581 2.64844 0.0519482 0.0430294 2024-09-22T03:12:05.972 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 14 16 5185 5169 2.88427 2.73438 0.0398867 0.0432018 2024-09-22T03:12:06.010 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [2, 0, 1, 3] 2024-09-22T03:12:06.010 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-22T03:12:06.010 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-09-22T03:12:06.010 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-09-22T03:12:06.160 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:12:06.153+0000 7fc348797640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:12:06.165 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:12:06.163+0000 7fc348797640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:12:06.206 INFO:teuthology.orchestra.run.smithi153.stderr:no valid command found; 10 closest matches: 2024-09-22T03:12:06.206 INFO:teuthology.orchestra.run.smithi153.stderr:config get 2024-09-22T03:12:06.206 INFO:teuthology.orchestra.run.smithi153.stderr:config help 2024-09-22T03:12:06.206 INFO:teuthology.orchestra.run.smithi153.stderr:config set ... 2024-09-22T03:12:06.206 INFO:teuthology.orchestra.run.smithi153.stderr:config show 2024-09-22T03:12:06.206 INFO:teuthology.orchestra.run.smithi153.stderr:deep_scrub 2024-09-22T03:12:06.206 INFO:teuthology.orchestra.run.smithi153.stderr:dump_historic_ops 2024-09-22T03:12:06.206 INFO:teuthology.orchestra.run.smithi153.stderr:dump_historic_slow_ops 2024-09-22T03:12:06.207 INFO:teuthology.orchestra.run.smithi153.stderr:dump_metrics [] 2024-09-22T03:12:06.207 INFO:teuthology.orchestra.run.smithi153.stderr:dump_ops_in_flight 2024-09-22T03:12:06.207 INFO:teuthology.orchestra.run.smithi153.stderr:dump_pgstate_history 2024-09-22T03:12:06.207 INFO:teuthology.orchestra.run.smithi153.stderr:Error EINVAL: invalid command 2024-09-22T03:12:06.210 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-22T03:12:06.210 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-22T03:12:06.210 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-09-22T03:12:06.259 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-22T03:12:06.340 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:12:06.333+0000 7f8482197640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:12:06.344 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:12:06.343+0000 7f8482197640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:12:06.365 INFO:teuthology.orchestra.run.smithi153.stderr:no valid command found; 10 closest matches: 2024-09-22T03:12:06.365 INFO:teuthology.orchestra.run.smithi153.stderr:config get 2024-09-22T03:12:06.365 INFO:teuthology.orchestra.run.smithi153.stderr:config help 2024-09-22T03:12:06.365 INFO:teuthology.orchestra.run.smithi153.stderr:config set ... 2024-09-22T03:12:06.365 INFO:teuthology.orchestra.run.smithi153.stderr:config show 2024-09-22T03:12:06.365 INFO:teuthology.orchestra.run.smithi153.stderr:deep_scrub 2024-09-22T03:12:06.365 INFO:teuthology.orchestra.run.smithi153.stderr:dump_historic_ops 2024-09-22T03:12:06.365 INFO:teuthology.orchestra.run.smithi153.stderr:dump_historic_slow_ops 2024-09-22T03:12:06.365 INFO:teuthology.orchestra.run.smithi153.stderr:dump_metrics [] 2024-09-22T03:12:06.365 INFO:teuthology.orchestra.run.smithi153.stderr:dump_ops_in_flight 2024-09-22T03:12:06.366 INFO:teuthology.orchestra.run.smithi153.stderr:dump_pgstate_history 2024-09-22T03:12:06.366 INFO:teuthology.orchestra.run.smithi153.stderr:Error EINVAL: invalid command 2024-09-22T03:12:06.368 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-22T03:12:06.368 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-22T03:12:06.369 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-09-22T03:12:06.405 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:12:06.402+0000 7fb9d0397640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:12:06.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:12:06.410+0000 7fb9d0397640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:12:06.509 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:12:06.503+0000 7fcdeef97640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:12:06.513 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:12:06.511+0000 7fcdeef97640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:12:06.557 INFO:teuthology.orchestra.run.smithi153.stderr:no valid command found; 10 closest matches: 2024-09-22T03:12:06.557 INFO:teuthology.orchestra.run.smithi153.stderr:config get 2024-09-22T03:12:06.557 INFO:teuthology.orchestra.run.smithi153.stderr:config help 2024-09-22T03:12:06.557 INFO:teuthology.orchestra.run.smithi153.stderr:config set ... 2024-09-22T03:12:06.558 INFO:teuthology.orchestra.run.smithi153.stderr:config show 2024-09-22T03:12:06.558 INFO:teuthology.orchestra.run.smithi153.stderr:deep_scrub 2024-09-22T03:12:06.558 INFO:teuthology.orchestra.run.smithi153.stderr:dump_historic_ops 2024-09-22T03:12:06.558 INFO:teuthology.orchestra.run.smithi153.stderr:dump_historic_slow_ops 2024-09-22T03:12:06.558 INFO:teuthology.orchestra.run.smithi153.stderr:dump_metrics [] 2024-09-22T03:12:06.558 INFO:teuthology.orchestra.run.smithi153.stderr:dump_ops_in_flight 2024-09-22T03:12:06.558 INFO:teuthology.orchestra.run.smithi153.stderr:dump_pgstate_history 2024-09-22T03:12:06.558 INFO:teuthology.orchestra.run.smithi153.stderr:Error EINVAL: invalid command 2024-09-22T03:12:06.561 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-22T03:12:06.562 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-22T03:12:06.562 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-09-22T03:12:06.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:12:06.732+0000 7fba25397640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:12:06.746 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:12:06.744+0000 7fba25397640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:12:06.763 INFO:teuthology.orchestra.run.smithi153.stderr:no valid command found; 10 closest matches: 2024-09-22T03:12:06.763 INFO:teuthology.orchestra.run.smithi153.stderr:config get 2024-09-22T03:12:06.763 INFO:teuthology.orchestra.run.smithi153.stderr:config help 2024-09-22T03:12:06.763 INFO:teuthology.orchestra.run.smithi153.stderr:config set ... 2024-09-22T03:12:06.763 INFO:teuthology.orchestra.run.smithi153.stderr:config show 2024-09-22T03:12:06.763 INFO:teuthology.orchestra.run.smithi153.stderr:deep_scrub 2024-09-22T03:12:06.763 INFO:teuthology.orchestra.run.smithi153.stderr:dump_historic_ops 2024-09-22T03:12:06.763 INFO:teuthology.orchestra.run.smithi153.stderr:dump_historic_slow_ops 2024-09-22T03:12:06.763 INFO:teuthology.orchestra.run.smithi153.stderr:dump_metrics [] 2024-09-22T03:12:06.764 INFO:teuthology.orchestra.run.smithi153.stderr:dump_ops_in_flight 2024-09-22T03:12:06.764 INFO:teuthology.orchestra.run.smithi153.stderr:dump_pgstate_history 2024-09-22T03:12:06.764 INFO:teuthology.orchestra.run.smithi153.stderr:Error EINVAL: invalid command 2024-09-22T03:12:06.767 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-22T03:12:06.767 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-22T03:12:06.972 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 15 16 5517 5501 2.86489 2.59375 0.0361905 0.0435423 2024-09-22T03:12:07.972 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 16 16 5860 5844 2.8533 2.67969 0.0276203 0.0437338 2024-09-22T03:12:08.526 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-22T03:12:08.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:12:08.664+0000 7fc996e63640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:12:08.674 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:12:08.672+0000 7fc996e63640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:12:08.972 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 17 16 6211 6195 2.84675 2.74219 0.0189418 0.0438485 2024-09-22T03:12:09.972 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 18 16 6546 6530 2.83399 2.61719 0.05878 0.0440083 2024-09-22T03:12:10.837 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-22T03:12:10.972 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 19 16 6900 6884 2.83038 2.76562 0.0252603 0.0441175 2024-09-22T03:12:10.978 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:12:10.974+0000 7f51aa8d7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:12:10.983 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:12:10.981+0000 7f51aa8d7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:12:11.768 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [2, 0, 1, 3] 2024-09-22T03:12:11.768 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-22T03:12:11.768 INFO:tasks.thrashosds.thrasher:Killing osd 1, live_osds are [2, 0, 1, 3] 2024-09-22T03:12:11.768 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-09-22T03:12:11.769 INFO:teuthology.orchestra.run:waiting for 300 2024-09-22T03:12:11.972 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:2024-09-22T03:12:11.971797+0000 min lat: 0.00928116 max lat: 0.128684 avg lat: 0.0442397 2024-09-22T03:12:11.972 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-22T03:12:11.972 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 20 16 7191 7175 2.80251 2.27344 0.0573293 0.0442397 2024-09-22T03:12:12.972 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 21 16 7191 7175 2.66905 0 - 0.0442397 2024-09-22T03:12:13.176 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-22T03:12:13.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:12:13.288+0000 7f2416e6c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:12:13.296 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:12:13.294+0000 7f2416e6c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:12:13.972 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 22 16 7191 7175 2.54772 0 - 0.0442397 2024-09-22T03:12:14.973 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 23 16 7191 7175 2.43694 0 - 0.0442397 2024-09-22T03:12:15.973 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 24 16 7191 7175 2.3354 0 - 0.0442397 2024-09-22T03:12:16.973 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 25 16 7191 7175 2.24198 0 - 0.0442397 2024-09-22T03:12:17.103 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~0s 2024-09-22T03:12:17.870 INFO:tasks.ceph.osd.1:Stopped 2024-09-22T03:12:17.973 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 26 16 7191 7175 2.15574 0 - 0.0442397 2024-09-22T03:12:18.973 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 27 16 7191 7175 2.0759 0 - 0.0442397 2024-09-22T03:12:19.973 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 28 16 7191 7175 2.00175 0 - 0.0442397 2024-09-22T03:12:20.974 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 29 16 7191 7175 1.93272 0 - 0.0442397 2024-09-22T03:12:21.974 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 30 16 7191 7175 1.8683 0 - 0.0442397 2024-09-22T03:12:22.607 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 has been restored 2024-09-22T03:12:22.871 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [1] live_osds: [2, 0, 3] 2024-09-22T03:12:22.871 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-22T03:12:22.871 INFO:tasks.thrashosds.thrasher:Reviving osd 1 2024-09-22T03:12:22.872 INFO:tasks.ceph.osd.1:Restarting daemon 2024-09-22T03:12:22.872 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2>> /var/log/ceph/ceph-osd.1.log 2024-09-22T03:12:22.874 INFO:tasks.ceph.osd.1:Started 2024-09-22T03:12:22.875 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-22T03:12:22.974 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 31 16 7191 7175 1.80803 0 - 0.0442397 2024-09-22T03:12:22.986 INFO:teuthology.orchestra.run.smithi153.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-09-22T03:12:22.989 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-22T03:12:22.990 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2024-09-22T03:12:23.974 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 32 16 7191 7175 1.75153 0 - 0.0442397 2024-09-22T03:12:24.974 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 33 16 7191 7175 1.69845 0 - 0.0442397 2024-09-22T03:12:25.974 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 34 16 7191 7175 1.6485 0 - 0.0442397 2024-09-22T03:12:26.974 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 35 16 7191 7175 1.6014 0 - 0.0442397 2024-09-22T03:12:27.974 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 36 16 7191 7175 1.55691 0 - 0.0442397 2024-09-22T03:12:27.990 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-22T03:12:28.087 INFO:teuthology.orchestra.run.smithi153.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-09-22T03:12:28.089 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-22T03:12:28.090 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2024-09-22T03:12:28.974 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 37 16 7191 7175 1.51483 0 - 0.0442397 2024-09-22T03:12:29.975 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 38 16 7191 7175 1.47497 0 - 0.0442397 2024-09-22T03:12:30.975 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 39 16 7191 7175 1.43715 0 - 0.0442397 2024-09-22T03:12:31.975 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:2024-09-22T03:12:31.974418+0000 min lat: 0.00928116 max lat: 0.128684 avg lat: 0.0442397 2024-09-22T03:12:31.975 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-22T03:12:31.975 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 40 16 7191 7175 1.40122 0 - 0.0442397 2024-09-22T03:12:32.975 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 41 16 7191 7175 1.36704 0 - 0.0442397 2024-09-22T03:12:33.091 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-22T03:12:33.193 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-22T03:12:33.193 INFO:teuthology.orchestra.run.smithi153.stdout: "ops_in_flight": [] 2024-09-22T03:12:33.193 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-22T03:12:33.884 INFO:teuthology.orchestra.run.smithi153.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-09-22T03:12:33.975 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 42 16 7191 7175 1.3345 0 - 0.0442397 2024-09-22T03:12:33.982 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-22T03:12:33.983 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-22T03:12:34.975 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 43 16 7191 7175 1.30346 0 - 0.0442397 2024-09-22T03:12:35.975 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 44 16 7191 7175 1.27384 0 - 0.0442397 2024-09-22T03:12:35.984 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-22T03:12:36.121 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:12:36.118+0000 7f19960d7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:12:36.125 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:12:36.123+0000 7f19960d7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:12:36.975 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 45 16 7511 7495 1.30108 0.1 0.0494217 0.0960291 2024-09-22T03:12:37.975 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 46 16 7848 7832 1.33003 2.63281 0.0504481 0.0939002 2024-09-22T03:12:38.203 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [2, 0, 3, 1] 2024-09-22T03:12:38.203 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-22T03:12:38.203 INFO:tasks.thrashosds.thrasher:Killing osd 3, live_osds are [2, 0, 3, 1] 2024-09-22T03:12:38.203 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-09-22T03:12:38.204 INFO:teuthology.orchestra.run:waiting for 300 2024-09-22T03:12:38.245 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-22T03:12:38.349 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:12:38.346+0000 7f8eb222c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:12:38.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:12:38.350+0000 7f8eb222c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:12:38.975 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 47 16 7936 7920 1.31635 0.6875 0.0591652 0.0933681 2024-09-22T03:12:39.322 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~0s 2024-09-22T03:12:39.976 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 48 16 7936 7920 1.28893 0 - 0.0933681 2024-09-22T03:12:40.976 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 49 16 7936 7920 1.26262 0 - 0.0933681 2024-09-22T03:12:41.976 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 50 16 7936 7920 1.23737 0 - 0.0933681 2024-09-22T03:12:42.976 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 51 16 7936 7920 1.21311 0 - 0.0933681 2024-09-22T03:12:43.976 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 52 16 7936 7920 1.18978 0 - 0.0933681 2024-09-22T03:12:44.304 INFO:tasks.ceph.osd.3:Stopped 2024-09-22T03:12:44.826 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 has been restored 2024-09-22T03:12:44.976 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 53 16 7936 7920 1.16733 0 - 0.0933681 2024-09-22T03:12:45.976 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 54 16 7936 7920 1.14571 0 - 0.0933681 2024-09-22T03:12:46.976 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 55 16 7936 7920 1.12488 0 - 0.0933681 2024-09-22T03:12:47.978 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 56 16 7936 7920 1.10479 0 - 0.0933681 2024-09-22T03:12:48.977 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 57 16 7936 7920 1.08541 0 - 0.0933681 2024-09-22T03:12:49.305 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [3] live_osds: [2, 0, 1] 2024-09-22T03:12:49.305 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-22T03:12:49.305 INFO:tasks.thrashosds.thrasher:Killing osd 0, live_osds are [2, 0, 1] 2024-09-22T03:12:49.306 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-09-22T03:12:49.306 INFO:teuthology.orchestra.run:waiting for 300 2024-09-22T03:12:49.405 INFO:tasks.ceph.osd.0:Stopped 2024-09-22T03:12:49.977 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 58 16 7936 7920 1.0667 0 - 0.0933681 2024-09-22T03:12:50.977 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 59 16 7936 7920 1.04862 0 - 0.0933681 2024-09-22T03:12:51.977 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:2024-09-22T03:12:51.976738+0000 min lat: 0.00928116 max lat: 24.3324 avg lat: 0.0933681 2024-09-22T03:12:51.977 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-22T03:12:51.977 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 60 16 7936 7920 1.03114 0 - 0.0933681 2024-09-22T03:12:52.977 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 61 16 7936 7920 1.01423 0 - 0.0933681 2024-09-22T03:12:53.977 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 62 16 7936 7920 0.997874 0 - 0.0933681 2024-09-22T03:12:54.406 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [3, 0] live_osds: [2, 1] 2024-09-22T03:12:54.406 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-22T03:12:54.406 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-09-22T03:12:54.407 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-09-22T03:12:54.527 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:12:54.520+0000 7fc4d1d97640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:12:54.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:12:54.528+0000 7fc4d1d97640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:12:54.977 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 63 16 7936 7920 0.982035 0 - 0.0933681 2024-09-22T03:12:55.978 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 64 16 7936 7920 0.96669 0 - 0.0933681 2024-09-22T03:12:56.978 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 65 16 7936 7920 0.951817 0 - 0.0933681 2024-09-22T03:12:57.978 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 66 16 7936 7920 0.937395 0 - 0.0933681 2024-09-22T03:12:58.978 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 67 16 7936 7920 0.923404 0 - 0.0933681 2024-09-22T03:12:59.978 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 68 16 7936 7920 0.909824 0 - 0.0933681 2024-09-22T03:13:00.979 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 69 16 7936 7920 0.896638 0 - 0.0933681 2024-09-22T03:13:01.979 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 70 16 7936 7920 0.883828 0 - 0.0933681 2024-09-22T03:13:02.979 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 71 16 7936 7920 0.871379 0 - 0.0933681 2024-09-22T03:13:03.979 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 72 16 7936 7920 0.859276 0 - 0.0933681 2024-09-22T03:13:04.979 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 73 16 7936 7920 0.847505 0 - 0.0933681 2024-09-22T03:13:05.897 INFO:teuthology.orchestra.run.smithi153.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-09-22T03:13:05.901 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-22T03:13:05.901 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-22T03:13:05.979 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 74 16 7936 7920 0.836052 0 - 0.0933681 2024-09-22T03:13:06.979 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 75 16 7936 7920 0.824904 0 - 0.0933681 2024-09-22T03:13:07.902 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-22T03:13:07.979 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 76 16 7936 7920 0.81405 0 - 0.0933681 2024-09-22T03:13:08.018 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:13:08.015+0000 7fe713d97640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:13:08.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:13:08.019+0000 7fe713d97640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:13:08.979 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 77 16 7936 7920 0.803478 0 - 0.0933681 2024-09-22T03:13:09.980 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 78 16 7936 7920 0.793177 0 - 0.0933681 2024-09-22T03:13:10.980 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 79 16 7936 7920 0.783137 0 - 0.0933681 2024-09-22T03:13:11.980 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:2024-09-22T03:13:11.979570+0000 min lat: 0.00928116 max lat: 24.3324 avg lat: 0.0933681 2024-09-22T03:13:11.980 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-22T03:13:11.980 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 80 16 7936 7920 0.773347 0 - 0.0933681 2024-09-22T03:13:12.980 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 81 16 7936 7920 0.763799 0 - 0.0933681 2024-09-22T03:13:13.980 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 82 16 7936 7920 0.754484 0 - 0.0933681 2024-09-22T03:13:14.980 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 83 16 7936 7920 0.745394 0 - 0.0933681 2024-09-22T03:13:15.981 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 84 16 7936 7920 0.73652 0 - 0.0933681 2024-09-22T03:13:16.981 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 85 16 7936 7920 0.727855 0 - 0.0933681 2024-09-22T03:13:17.231 INFO:teuthology.orchestra.run.smithi153.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-22T03:13:17.231 INFO:teuthology.orchestra.run.smithi153.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-22T03:13:17.235 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-22T03:13:17.236 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-22T03:13:17.236 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-09-22T03:13:17.297 INFO:teuthology.orchestra.run.smithi153.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-09-22T03:13:17.300 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-22T03:13:17.301 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-22T03:13:17.358 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:13:17.351+0000 7f95d54ce640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:13:17.360 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:13:17.359+0000 7f95d54ce640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:13:17.372 INFO:teuthology.orchestra.run.smithi153.stderr:no valid command found; 10 closest matches: 2024-09-22T03:13:17.372 INFO:teuthology.orchestra.run.smithi153.stderr:config get 2024-09-22T03:13:17.372 INFO:teuthology.orchestra.run.smithi153.stderr:config help 2024-09-22T03:13:17.372 INFO:teuthology.orchestra.run.smithi153.stderr:config set ... 2024-09-22T03:13:17.372 INFO:teuthology.orchestra.run.smithi153.stderr:config show 2024-09-22T03:13:17.373 INFO:teuthology.orchestra.run.smithi153.stderr:deep_scrub 2024-09-22T03:13:17.373 INFO:teuthology.orchestra.run.smithi153.stderr:dump_historic_ops 2024-09-22T03:13:17.373 INFO:teuthology.orchestra.run.smithi153.stderr:dump_historic_slow_ops 2024-09-22T03:13:17.373 INFO:teuthology.orchestra.run.smithi153.stderr:dump_metrics [] 2024-09-22T03:13:17.373 INFO:teuthology.orchestra.run.smithi153.stderr:dump_ops_in_flight 2024-09-22T03:13:17.373 INFO:teuthology.orchestra.run.smithi153.stderr:dump_pgstate_history 2024-09-22T03:13:17.373 INFO:teuthology.orchestra.run.smithi153.stderr:Error EINVAL: invalid command 2024-09-22T03:13:17.375 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-22T03:13:17.376 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-22T03:13:17.376 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-09-22T03:13:17.481 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:13:17.475+0000 7fdf1e46c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:13:17.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:13:17.483+0000 7fdf1e46c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:13:17.497 INFO:teuthology.orchestra.run.smithi153.stderr:no valid command found; 10 closest matches: 2024-09-22T03:13:17.497 INFO:teuthology.orchestra.run.smithi153.stderr:config get 2024-09-22T03:13:17.497 INFO:teuthology.orchestra.run.smithi153.stderr:config help 2024-09-22T03:13:17.497 INFO:teuthology.orchestra.run.smithi153.stderr:config set ... 2024-09-22T03:13:17.497 INFO:teuthology.orchestra.run.smithi153.stderr:config show 2024-09-22T03:13:17.497 INFO:teuthology.orchestra.run.smithi153.stderr:deep_scrub 2024-09-22T03:13:17.497 INFO:teuthology.orchestra.run.smithi153.stderr:dump_historic_ops 2024-09-22T03:13:17.497 INFO:teuthology.orchestra.run.smithi153.stderr:dump_historic_slow_ops 2024-09-22T03:13:17.497 INFO:teuthology.orchestra.run.smithi153.stderr:dump_metrics [] 2024-09-22T03:13:17.498 INFO:teuthology.orchestra.run.smithi153.stderr:dump_ops_in_flight 2024-09-22T03:13:17.498 INFO:teuthology.orchestra.run.smithi153.stderr:dump_pgstate_history 2024-09-22T03:13:17.498 INFO:teuthology.orchestra.run.smithi153.stderr:Error EINVAL: invalid command 2024-09-22T03:13:17.501 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-22T03:13:17.502 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-22T03:13:17.502 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-09-22T03:13:17.606 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:13:17.600+0000 7fcfbd397640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:13:17.609 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:13:17.608+0000 7fcfbd397640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:13:17.613 INFO:teuthology.orchestra.run.smithi153.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-09-22T03:13:17.616 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-22T03:13:17.616 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-22T03:13:17.981 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 86 16 7936 7920 0.719391 0 - 0.0933681 2024-09-22T03:13:18.981 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 87 16 7939 7923 0.711391 0.000585937 0.0247281 0.103454 2024-09-22T03:13:19.302 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-22T03:13:19.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:13:19.413+0000 7f62a84d7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:13:19.420 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:13:19.418+0000 7f62a84d7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:13:19.423 INFO:teuthology.orchestra.run.smithi153.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-22T03:13:19.452 INFO:teuthology.orchestra.run.smithi153.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-09-22T03:13:19.455 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-22T03:13:19.456 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-22T03:13:19.981 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 88 16 7939 7923 0.703307 0 - 0.103454 2024-09-22T03:13:20.981 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 89 16 7939 7923 0.695404 0 - 0.103454 2024-09-22T03:13:21.457 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-22T03:13:21.572 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:13:21.569+0000 7f4078197640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:13:21.575 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:13:21.574+0000 7f4078197640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:13:21.579 INFO:teuthology.orchestra.run.smithi153.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-22T03:13:21.608 INFO:teuthology.orchestra.run.smithi153.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-09-22T03:13:21.611 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-22T03:13:21.611 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-22T03:13:21.981 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 90 16 7939 7923 0.687677 0 - 0.103454 2024-09-22T03:13:22.617 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [3, 0] live_osds: [2, 1] 2024-09-22T03:13:22.617 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-22T03:13:22.618 INFO:tasks.thrashosds.thrasher:Reviving osd 3 2024-09-22T03:13:22.618 INFO:tasks.ceph.osd.3:Restarting daemon 2024-09-22T03:13:22.618 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2>> /var/log/ceph/ceph-osd.3.log 2024-09-22T03:13:22.621 INFO:tasks.ceph.osd.3:Started 2024-09-22T03:13:22.622 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-22T03:13:22.732 INFO:teuthology.orchestra.run.smithi196.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-09-22T03:13:22.735 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-22T03:13:22.736 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-09-22T03:13:22.982 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 91 16 7939 7923 0.68012 0 - 0.103454 2024-09-22T03:13:23.612 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-22T03:13:23.728 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:13:23.725+0000 7fc930917640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:13:23.731 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:13:23.730+0000 7fc930917640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:13:23.735 INFO:teuthology.orchestra.run.smithi153.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-22T03:13:23.762 INFO:teuthology.orchestra.run.smithi153.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-09-22T03:13:23.765 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-22T03:13:23.765 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-22T03:13:23.982 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 92 16 7939 7923 0.672728 0 - 0.103454 2024-09-22T03:13:24.982 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 93 16 7939 7923 0.665494 0 - 0.103454 2024-09-22T03:13:25.767 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-22T03:13:25.882 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:13:25.879+0000 7f1036917640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:13:25.887 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:13:25.885+0000 7f1036917640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:13:25.890 INFO:teuthology.orchestra.run.smithi153.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-22T03:13:25.917 INFO:teuthology.orchestra.run.smithi153.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-09-22T03:13:25.920 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-22T03:13:25.921 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-22T03:13:25.982 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 94 16 7939 7923 0.658414 0 - 0.103454 2024-09-22T03:13:26.982 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 95 16 7939 7923 0.651484 0 - 0.103454 2024-09-22T03:13:27.736 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-22T03:13:27.832 INFO:teuthology.orchestra.run.smithi196.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-09-22T03:13:27.835 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-22T03:13:27.835 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-09-22T03:13:27.922 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-22T03:13:27.982 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 96 16 7939 7923 0.644697 0 - 0.103454 2024-09-22T03:13:28.040 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:13:28.037+0000 7f4974c6c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:13:28.043 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:13:28.042+0000 7f4974c6c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:13:28.049 INFO:teuthology.orchestra.run.smithi153.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-22T03:13:28.074 INFO:teuthology.orchestra.run.smithi153.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-09-22T03:13:28.077 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-22T03:13:28.077 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-22T03:13:28.983 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 97 16 7939 7923 0.638051 0 - 0.103454 2024-09-22T03:13:29.983 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 98 16 7939 7923 0.63154 0 - 0.103454 2024-09-22T03:13:30.078 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-22T03:13:30.186 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:13:30.183+0000 7fa467b97640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:13:30.189 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:13:30.187+0000 7fa467b97640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:13:30.192 INFO:teuthology.orchestra.run.smithi153.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-22T03:13:30.235 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-22T03:13:30.235 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-22T03:13:30.983 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 99 16 7939 7923 0.625161 0 - 0.103454 2024-09-22T03:13:31.983 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:Total time run: 99.0726 2024-09-22T03:13:31.983 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:Total writes made: 7939 2024-09-22T03:13:31.983 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:Write size: 8192 2024-09-22T03:13:31.983 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:Object size: 65536 2024-09-22T03:13:31.983 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:Bandwidth (MB/sec): 0.62604 2024-09-22T03:13:31.983 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:Stddev Bandwidth: 1.14966 2024-09-22T03:13:31.983 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:Max bandwidth (MB/sec): 3.16406 2024-09-22T03:13:31.983 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:Min bandwidth (MB/sec): 0 2024-09-22T03:13:31.983 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:Average IOPS: 80 2024-09-22T03:13:31.984 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:Stddev IOPS: 147.16 2024-09-22T03:13:31.984 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:Max IOPS: 405 2024-09-22T03:13:31.984 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:Min IOPS: 0 2024-09-22T03:13:31.984 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:Average Latency(s): 0.199618 2024-09-22T03:13:31.984 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:Stddev Latency(s): 2.5524 2024-09-22T03:13:31.984 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:Max latency(s): 52.8389 2024-09-22T03:13:31.984 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:Min latency(s): 0.00928116 2024-09-22T03:13:32.002 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:Cleaning up (deleting benchmark objects) 2024-09-22T03:13:32.236 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-22T03:13:32.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:13:32.363+0000 7f7822397640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:13:32.370 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:13:32.368+0000 7f7822397640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:13:32.374 INFO:teuthology.orchestra.run.smithi153.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-22T03:13:32.535 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-22T03:13:32.536 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-22T03:13:32.836 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-22T03:13:33.010 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-09-22T03:13:33.010 INFO:teuthology.orchestra.run.smithi196.stdout: "ops_in_flight": [ 2024-09-22T03:13:33.010 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-22T03:13:33.010 INFO:teuthology.orchestra.run.smithi196.stdout: "type": "client_request", 2024-09-22T03:13:33.011 INFO:teuthology.orchestra.run.smithi196.stdout: "id": 335, 2024-09-22T03:13:33.011 INFO:teuthology.orchestra.run.smithi196.stdout: "detail": { 2024-09-22T03:13:33.011 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.011 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:13:33.011 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.601664+0000" 2024-09-22T03:13:33.011 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.011 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.011 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:13:33.011 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.601671+0000" 2024-09-22T03:13:33.011 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.011 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.011 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.011 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.012 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.012 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.012 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:13:33.012 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.602057+0000" 2024-09-22T03:13:33.012 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.012 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.012 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.012 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.012 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.012 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.012 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:13:33.012 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.012 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.012 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.013 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.013 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.013 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.013 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.013 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:13:33.013 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.602870+0000" 2024-09-22T03:13:33.013 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.013 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.013 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.013 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.013 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.013 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.013 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:13:33.013 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.602919+0000" 2024-09-22T03:13:33.013 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.014 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.014 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.014 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.014 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.014 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.014 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:13:33.014 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.014 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.014 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.014 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.014 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.014 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.014 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.014 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:13:33.015 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.644080+0000" 2024-09-22T03:13:33.015 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.015 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.015 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:13:33.015 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.603119+0000" 2024-09-22T03:13:33.015 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.015 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.015 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.015 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.015 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.015 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.015 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:13:33.015 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.015 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.016 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.016 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.016 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.016 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.016 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.016 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:13:33.016 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.603215+0000" 2024-09-22T03:13:33.016 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.016 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.016 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.016 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.016 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.016 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.016 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:13:33.016 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.017 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.017 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.017 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.017 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.017 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.017 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.017 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:13:33.017 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.603319+0000" 2024-09-22T03:13:33.017 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.017 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.017 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.017 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.017 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.017 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.018 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:13:33.018 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.018 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.018 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.018 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.018 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.018 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.018 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.018 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:13:33.018 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.018 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.018 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.018 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.018 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.019 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.019 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.019 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:13:33.019 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.019 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.019 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.019 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.019 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.019 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.019 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.019 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:13:33.019 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.603554+0000" 2024-09-22T03:13:33.019 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.019 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.019 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.020 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.020 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.020 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.020 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:13:33.020 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.020 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.020 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.020 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.020 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.020 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.020 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.020 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:13:33.020 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.606541+0000" 2024-09-22T03:13:33.020 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.021 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.021 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.021 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.021 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.021 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.021 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:13:33.021 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.021 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.021 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.021 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.021 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.021 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.021 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.021 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:13:33.022 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.643636+0000" 2024-09-22T03:13:33.022 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.022 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.022 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.022 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.022 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.022 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.022 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:13:33.022 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.022 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:13:33.022 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:13:33.022 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.022 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-22T03:13:33.022 INFO:teuthology.orchestra.run.smithi196.stdout: "type": "client_request", 2024-09-22T03:13:33.023 INFO:teuthology.orchestra.run.smithi196.stdout: "id": 343, 2024-09-22T03:13:33.023 INFO:teuthology.orchestra.run.smithi196.stdout: "detail": { 2024-09-22T03:13:33.023 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.023 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:13:33.023 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.625657+0000" 2024-09-22T03:13:33.023 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.023 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.023 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:13:33.023 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.625662+0000" 2024-09-22T03:13:33.023 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.023 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.023 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.023 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.023 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.023 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.024 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:13:33.024 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.625942+0000" 2024-09-22T03:13:33.024 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.024 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.024 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.024 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.024 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.024 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.024 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:13:33.024 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.024 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.024 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.024 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.024 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.024 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.024 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.025 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:13:33.025 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.626097+0000" 2024-09-22T03:13:33.025 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.025 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.025 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.025 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.025 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.025 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.025 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:13:33.025 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.626442+0000" 2024-09-22T03:13:33.025 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.025 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.025 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.025 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.025 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.026 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.026 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:13:33.026 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.026 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.026 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.026 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.026 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.026 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.026 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.026 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:13:33.026 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.645948+0000" 2024-09-22T03:13:33.026 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.026 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.026 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:13:33.026 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.628510+0000" 2024-09-22T03:13:33.027 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.027 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.027 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.027 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.027 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.027 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.027 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:13:33.027 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.027 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.027 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.027 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.027 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.027 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.027 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.027 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:13:33.028 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.628623+0000" 2024-09-22T03:13:33.028 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.028 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.028 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.028 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.028 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.028 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.028 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:13:33.028 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.028 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.028 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.028 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.028 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.028 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.028 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.029 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:13:33.029 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.628750+0000" 2024-09-22T03:13:33.029 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.029 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.029 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.029 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.029 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.029 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.029 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:13:33.029 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.029 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.029 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.029 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.029 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.029 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.030 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.030 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:13:33.030 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.030 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.030 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.030 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.030 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.030 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.030 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.030 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:13:33.030 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.030 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.030 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.031 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.031 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.031 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.031 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.031 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:13:33.031 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.628937+0000" 2024-09-22T03:13:33.031 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.031 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.032 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.032 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.032 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.032 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.032 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:13:33.032 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.032 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.032 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.032 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.032 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.033 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.033 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.033 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:13:33.033 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.632887+0000" 2024-09-22T03:13:33.033 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.033 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.033 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.033 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.033 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.034 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.034 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:13:33.034 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.034 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.034 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.034 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.034 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.034 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.034 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.034 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:13:33.034 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.645526+0000" 2024-09-22T03:13:33.034 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.035 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.035 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.035 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.035 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.035 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.035 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:13:33.035 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.035 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:13:33.035 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:13:33.035 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.036 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-22T03:13:33.036 INFO:teuthology.orchestra.run.smithi196.stdout: "type": "client_request", 2024-09-22T03:13:33.036 INFO:teuthology.orchestra.run.smithi196.stdout: "id": 358, 2024-09-22T03:13:33.036 INFO:teuthology.orchestra.run.smithi196.stdout: "detail": { 2024-09-22T03:13:33.036 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.036 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:13:33.036 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.676862+0000" 2024-09-22T03:13:33.036 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.036 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.036 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:13:33.036 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.676873+0000" 2024-09-22T03:13:33.036 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.036 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.037 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.037 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.037 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.037 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.037 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:13:33.037 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.677244+0000" 2024-09-22T03:13:33.037 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.037 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.037 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.037 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.037 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.038 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.038 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:13:33.038 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.038 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.038 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.038 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.038 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.038 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.038 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.038 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:13:33.039 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.677333+0000" 2024-09-22T03:13:33.039 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.039 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.039 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.039 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.039 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.039 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.039 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:13:33.039 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.677434+0000" 2024-09-22T03:13:33.039 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.039 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.039 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.039 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.040 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.040 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.040 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:13:33.040 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.040 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.040 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.040 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.040 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.040 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.040 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.040 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:13:33.040 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.700319+0000" 2024-09-22T03:13:33.040 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.041 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.041 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:13:33.041 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.677650+0000" 2024-09-22T03:13:33.041 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.041 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.041 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.041 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.041 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.042 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.042 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:13:33.042 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.042 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.042 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.042 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.042 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.042 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.042 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.043 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:13:33.043 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.677760+0000" 2024-09-22T03:13:33.043 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.043 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.043 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.043 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.043 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.043 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.043 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:13:33.043 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.043 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.043 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.043 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.043 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.044 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.044 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.044 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:13:33.044 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.677877+0000" 2024-09-22T03:13:33.044 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.044 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.044 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.044 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.044 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.045 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.045 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:13:33.045 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.045 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.045 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.045 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.045 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.045 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.045 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.045 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:13:33.045 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.045 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.046 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.046 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.046 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.046 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.046 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.046 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:13:33.046 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.046 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.046 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.046 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.046 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.046 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.046 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.047 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:13:33.047 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.678666+0000" 2024-09-22T03:13:33.047 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.047 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.047 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.047 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.047 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.047 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.047 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:13:33.047 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.047 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.047 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.047 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.048 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.048 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.048 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.048 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:13:33.048 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.680558+0000" 2024-09-22T03:13:33.048 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.048 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.048 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.048 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.048 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.048 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.048 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:13:33.048 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.048 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.048 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.049 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.049 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.049 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.049 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.049 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:13:33.049 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.699006+0000" 2024-09-22T03:13:33.049 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.049 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.049 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.049 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.049 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.049 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.049 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:13:33.049 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.050 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:13:33.050 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:13:33.050 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.050 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-22T03:13:33.050 INFO:teuthology.orchestra.run.smithi196.stdout: "type": "client_request", 2024-09-22T03:13:33.050 INFO:teuthology.orchestra.run.smithi196.stdout: "id": 368, 2024-09-22T03:13:33.050 INFO:teuthology.orchestra.run.smithi196.stdout: "detail": { 2024-09-22T03:13:33.050 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.050 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:13:33.050 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.708117+0000" 2024-09-22T03:13:33.050 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.051 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.051 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:13:33.051 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.708124+0000" 2024-09-22T03:13:33.051 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.051 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.051 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.051 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.051 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.051 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.051 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:13:33.051 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.708253+0000" 2024-09-22T03:13:33.051 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.051 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.052 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.052 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.052 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.052 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.052 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:13:33.052 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.052 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.052 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.052 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.052 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.052 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.052 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.052 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:13:33.052 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.708330+0000" 2024-09-22T03:13:33.052 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.053 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.053 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.053 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.053 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.053 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.053 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:13:33.053 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.708409+0000" 2024-09-22T03:13:33.053 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.053 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.053 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.053 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.053 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.053 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.053 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:13:33.054 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.054 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.054 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.054 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.054 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.054 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.054 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.054 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:13:33.054 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.723288+0000" 2024-09-22T03:13:33.054 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.054 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.054 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:13:33.054 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.708621+0000" 2024-09-22T03:13:33.055 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.055 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.055 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.055 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.055 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.055 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.055 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:13:33.055 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.055 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.055 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.055 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.055 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.055 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.055 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.056 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:13:33.056 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.708706+0000" 2024-09-22T03:13:33.056 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.056 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.056 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.056 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.056 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.056 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.056 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:13:33.056 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.056 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.056 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.056 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.056 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.056 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.057 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.057 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:13:33.057 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.708818+0000" 2024-09-22T03:13:33.057 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.057 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.057 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.057 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.057 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.057 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.057 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:13:33.057 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.057 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.057 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.057 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.058 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.058 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.058 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.058 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:13:33.058 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.058 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.058 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.058 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.058 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.058 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.058 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.058 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:13:33.058 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.058 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.059 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.059 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.059 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.059 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.059 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.059 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:13:33.059 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.708956+0000" 2024-09-22T03:13:33.059 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.059 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.059 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.059 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.059 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.059 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.059 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:13:33.060 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.060 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.060 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.060 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.060 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.060 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.060 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.060 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:13:33.060 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.710589+0000" 2024-09-22T03:13:33.060 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.060 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.060 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.060 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.060 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.060 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.061 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:13:33.061 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.061 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.061 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.061 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.061 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.061 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.061 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.061 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:13:33.061 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.722701+0000" 2024-09-22T03:13:33.061 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.061 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.061 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.061 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.062 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.062 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.062 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:13:33.062 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.062 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:13:33.062 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:13:33.062 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.062 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-22T03:13:33.062 INFO:teuthology.orchestra.run.smithi196.stdout: "type": "client_request", 2024-09-22T03:13:33.062 INFO:teuthology.orchestra.run.smithi196.stdout: "id": 369, 2024-09-22T03:13:33.062 INFO:teuthology.orchestra.run.smithi196.stdout: "detail": { 2024-09-22T03:13:33.062 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.062 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:13:33.063 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.713590+0000" 2024-09-22T03:13:33.063 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.063 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.063 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:13:33.063 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.713597+0000" 2024-09-22T03:13:33.063 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.063 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.063 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.063 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.063 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.063 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.063 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:13:33.063 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.713676+0000" 2024-09-22T03:13:33.063 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.063 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.064 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.064 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.064 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.064 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.064 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:13:33.064 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.064 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.064 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.064 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.064 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.064 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.064 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.064 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:13:33.064 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.713750+0000" 2024-09-22T03:13:33.065 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.065 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.065 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.065 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.065 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.065 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.065 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:13:33.065 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.713838+0000" 2024-09-22T03:13:33.065 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.065 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.065 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.065 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.065 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.065 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.066 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:13:33.066 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.066 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.066 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.066 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.066 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.066 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.066 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.066 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:13:33.066 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.735860+0000" 2024-09-22T03:13:33.066 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.066 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.066 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:13:33.067 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.714034+0000" 2024-09-22T03:13:33.067 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.067 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.067 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.067 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.067 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.067 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.067 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:13:33.067 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.067 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.067 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.068 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.068 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.068 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.068 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.068 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:13:33.068 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.714140+0000" 2024-09-22T03:13:33.068 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.068 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.068 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.068 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.068 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.068 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.068 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:13:33.068 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.069 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.069 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.069 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.069 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.069 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.069 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.069 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:13:33.069 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.714238+0000" 2024-09-22T03:13:33.069 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.069 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.069 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.069 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.069 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.069 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.070 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:13:33.070 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.070 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.070 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.070 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.070 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.070 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.070 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.070 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:13:33.070 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.070 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.070 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.070 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.070 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.071 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.071 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.071 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:13:33.071 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.071 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.071 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.071 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.071 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.071 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.071 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.071 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:13:33.071 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.714809+0000" 2024-09-22T03:13:33.071 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.071 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.072 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.072 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.072 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.072 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.072 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:13:33.072 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.072 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.072 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.072 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.072 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.072 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.072 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.072 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:13:33.072 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.716347+0000" 2024-09-22T03:13:33.073 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.073 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.073 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.073 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.073 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.073 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.073 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:13:33.073 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.073 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.073 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.073 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.073 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.073 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.073 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.073 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:13:33.074 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.733821+0000" 2024-09-22T03:13:33.074 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.074 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.074 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.074 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.074 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.074 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.074 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:13:33.074 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.074 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:13:33.074 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:13:33.074 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.074 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-22T03:13:33.075 INFO:teuthology.orchestra.run.smithi196.stdout: "type": "client_request", 2024-09-22T03:13:33.075 INFO:teuthology.orchestra.run.smithi196.stdout: "id": 370, 2024-09-22T03:13:33.075 INFO:teuthology.orchestra.run.smithi196.stdout: "detail": { 2024-09-22T03:13:33.075 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.075 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:13:33.075 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.718827+0000" 2024-09-22T03:13:33.075 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.075 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.075 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:13:33.075 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.718835+0000" 2024-09-22T03:13:33.075 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.075 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.075 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.075 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.076 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.076 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.076 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:13:33.076 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.719004+0000" 2024-09-22T03:13:33.076 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.076 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.076 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.076 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.076 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.076 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.076 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:13:33.076 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.076 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.076 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.076 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.077 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.077 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.077 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.077 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:13:33.077 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.719190+0000" 2024-09-22T03:13:33.077 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.077 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.077 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.077 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.077 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.077 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.077 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:13:33.077 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.719354+0000" 2024-09-22T03:13:33.077 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.078 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.078 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.078 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.078 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.078 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.078 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:13:33.078 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.078 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.078 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.078 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.078 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.078 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.078 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.079 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:13:33.079 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.746721+0000" 2024-09-22T03:13:33.079 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.079 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.079 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:13:33.079 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.719679+0000" 2024-09-22T03:13:33.079 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.079 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.079 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.079 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.079 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.079 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.079 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:13:33.079 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.079 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.080 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.080 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.080 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.080 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.080 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.080 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:13:33.080 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.719848+0000" 2024-09-22T03:13:33.080 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.080 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.080 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.080 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.080 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.080 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.080 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:13:33.081 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.081 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.081 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.081 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.081 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.081 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.081 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.081 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:13:33.081 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.720292+0000" 2024-09-22T03:13:33.081 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.081 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.081 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.081 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.082 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.082 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.082 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:13:33.082 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.082 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.082 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.082 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.082 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.082 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.082 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.082 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:13:33.082 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.082 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.082 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.082 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.083 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.083 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.083 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.083 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:13:33.083 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.083 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.083 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.083 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.083 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.083 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.083 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.083 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:13:33.083 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.720900+0000" 2024-09-22T03:13:33.083 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.083 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.084 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.084 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.084 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.084 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.084 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:13:33.084 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.084 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.084 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.084 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.084 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.084 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.084 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.084 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:13:33.085 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.723819+0000" 2024-09-22T03:13:33.085 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.085 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.085 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.085 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.085 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.085 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.085 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:13:33.085 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.085 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.085 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.085 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.085 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.086 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.086 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.086 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:13:33.086 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.745904+0000" 2024-09-22T03:13:33.086 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.086 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.086 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.086 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.086 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.086 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.086 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:13:33.086 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.086 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:13:33.086 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:13:33.087 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.087 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-22T03:13:33.087 INFO:teuthology.orchestra.run.smithi196.stdout: "type": "client_request", 2024-09-22T03:13:33.087 INFO:teuthology.orchestra.run.smithi196.stdout: "id": 372, 2024-09-22T03:13:33.087 INFO:teuthology.orchestra.run.smithi196.stdout: "detail": { 2024-09-22T03:13:33.087 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.087 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:13:33.087 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.728291+0000" 2024-09-22T03:13:33.087 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.087 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.087 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:13:33.087 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.728299+0000" 2024-09-22T03:13:33.087 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.087 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.087 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.088 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.088 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.088 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.088 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:13:33.088 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.729423+0000" 2024-09-22T03:13:33.088 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.088 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.088 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.088 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.088 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.088 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.088 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:13:33.088 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.088 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.089 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.089 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.089 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.089 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.089 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.089 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:13:33.089 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.729644+0000" 2024-09-22T03:13:33.089 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.089 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.089 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.089 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.089 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.089 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.089 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:13:33.090 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.730513+0000" 2024-09-22T03:13:33.090 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.090 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.090 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.090 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.090 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.090 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.090 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:13:33.090 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.090 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.090 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.090 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.090 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.090 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.091 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.091 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:13:33.091 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.765126+0000" 2024-09-22T03:13:33.091 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.091 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.091 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:13:33.091 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.730815+0000" 2024-09-22T03:13:33.091 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.091 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.091 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.091 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.091 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.091 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.092 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:13:33.092 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.092 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.092 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.092 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.092 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.092 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.092 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.092 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:13:33.092 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.730946+0000" 2024-09-22T03:13:33.092 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.092 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.092 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.092 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.093 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.093 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.093 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:13:33.093 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.093 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.093 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.093 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.093 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.093 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.093 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.093 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:13:33.093 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.732101+0000" 2024-09-22T03:13:33.093 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.093 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.094 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.094 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.094 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.094 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.094 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:13:33.094 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.094 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.094 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.094 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.094 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.094 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.094 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.094 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:13:33.094 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.095 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.095 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.095 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.095 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.095 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.095 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.095 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:13:33.095 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.095 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.095 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.095 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.095 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.095 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.095 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.096 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:13:33.096 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.732543+0000" 2024-09-22T03:13:33.096 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.096 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.096 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.096 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.096 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.096 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.096 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:13:33.096 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.096 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.096 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.096 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.096 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.097 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.097 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.097 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:13:33.097 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.736992+0000" 2024-09-22T03:13:33.097 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.097 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.097 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.097 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.097 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.097 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.097 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:13:33.097 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.097 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.097 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.098 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.098 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.098 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.098 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.098 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:13:33.098 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.763932+0000" 2024-09-22T03:13:33.098 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.098 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.098 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.098 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.098 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.098 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.098 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:13:33.098 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.099 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:13:33.099 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:13:33.099 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.099 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-22T03:13:33.099 INFO:teuthology.orchestra.run.smithi196.stdout: "type": "client_request", 2024-09-22T03:13:33.099 INFO:teuthology.orchestra.run.smithi196.stdout: "id": 375, 2024-09-22T03:13:33.099 INFO:teuthology.orchestra.run.smithi196.stdout: "detail": { 2024-09-22T03:13:33.099 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.099 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:13:33.099 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.741387+0000" 2024-09-22T03:13:33.099 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.099 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.099 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:13:33.099 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.741399+0000" 2024-09-22T03:13:33.100 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.100 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.100 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.100 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.100 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.100 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.100 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:13:33.100 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.742658+0000" 2024-09-22T03:13:33.100 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.100 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.100 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.100 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.100 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.100 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.101 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:13:33.101 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.101 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.101 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.101 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.101 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.101 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.101 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.101 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:13:33.101 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.742772+0000" 2024-09-22T03:13:33.101 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.101 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.101 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.101 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.101 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.102 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.102 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:13:33.102 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.742852+0000" 2024-09-22T03:13:33.102 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.102 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.102 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.102 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.102 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.102 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.102 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:13:33.102 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.103 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.103 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.103 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.103 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.103 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.103 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.103 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:13:33.103 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.771039+0000" 2024-09-22T03:13:33.103 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.103 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.103 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:13:33.103 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.743953+0000" 2024-09-22T03:13:33.103 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.103 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.104 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.104 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.104 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.104 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.104 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:13:33.104 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.104 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.104 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.104 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.104 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.104 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.104 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.104 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:13:33.105 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.744405+0000" 2024-09-22T03:13:33.105 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.105 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.105 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.105 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.105 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.105 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.105 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:13:33.105 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.105 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.105 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.105 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.105 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.105 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.105 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.106 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:13:33.106 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.744869+0000" 2024-09-22T03:13:33.106 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.106 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.106 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.106 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.106 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.106 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.106 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:13:33.106 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.106 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.106 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.106 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.106 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.107 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.107 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.107 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:13:33.107 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.107 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.107 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.107 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.107 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.107 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.107 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.107 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:13:33.107 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.107 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.107 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.107 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.108 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.108 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.108 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.108 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:13:33.108 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.745091+0000" 2024-09-22T03:13:33.108 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.108 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.108 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.108 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.108 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.108 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.108 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:13:33.108 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.108 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.109 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.109 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.109 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.109 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.109 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.109 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:13:33.109 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.747368+0000" 2024-09-22T03:13:33.109 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.109 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.109 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.109 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.109 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.109 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.110 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:13:33.110 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.110 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.110 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.110 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.110 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.110 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.110 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.110 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:13:33.110 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.770176+0000" 2024-09-22T03:13:33.110 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.110 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.110 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.110 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.111 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.111 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.111 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:13:33.111 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.111 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:13:33.111 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:13:33.111 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.111 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-22T03:13:33.111 INFO:teuthology.orchestra.run.smithi196.stdout: "type": "client_request", 2024-09-22T03:13:33.111 INFO:teuthology.orchestra.run.smithi196.stdout: "id": 384, 2024-09-22T03:13:33.111 INFO:teuthology.orchestra.run.smithi196.stdout: "detail": { 2024-09-22T03:13:33.111 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.111 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:13:33.112 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.770885+0000" 2024-09-22T03:13:33.112 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.112 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.112 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:13:33.112 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.770889+0000" 2024-09-22T03:13:33.112 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.112 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.112 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.112 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.112 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.112 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.112 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:13:33.112 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.771218+0000" 2024-09-22T03:13:33.113 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.113 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.113 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.113 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.113 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.113 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.113 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:13:33.113 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.113 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.113 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.113 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.113 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.113 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.113 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.113 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:13:33.114 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.771291+0000" 2024-09-22T03:13:33.114 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.114 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.114 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.114 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.114 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.114 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.114 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:13:33.114 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.771366+0000" 2024-09-22T03:13:33.114 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.114 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.114 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.114 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.115 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.115 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.115 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:13:33.115 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.115 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.115 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.115 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.115 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.115 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.115 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.115 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:13:33.115 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.810456+0000" 2024-09-22T03:13:33.115 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.116 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.116 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:13:33.116 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.771614+0000" 2024-09-22T03:13:33.116 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.116 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.116 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.116 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.116 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.116 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.116 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:13:33.116 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.116 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.116 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.116 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.116 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.117 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.117 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.117 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:13:33.117 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.771732+0000" 2024-09-22T03:13:33.117 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.117 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.117 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.117 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.117 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.117 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.117 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:13:33.117 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.117 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.117 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.118 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.118 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.118 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.118 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.118 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:13:33.118 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.771828+0000" 2024-09-22T03:13:33.118 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.118 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.118 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.118 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.118 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.118 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.118 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:13:33.119 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.119 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.119 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.119 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.119 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.119 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.119 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.119 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:13:33.119 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.119 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.119 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.120 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.120 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.120 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.120 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.120 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:13:33.120 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.120 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.120 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.120 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.120 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.120 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.120 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.120 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:13:33.120 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.772053+0000" 2024-09-22T03:13:33.120 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.121 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.121 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.121 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.121 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.121 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.121 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:13:33.121 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.121 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.121 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.121 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.121 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.121 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.121 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.121 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:13:33.121 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.778429+0000" 2024-09-22T03:13:33.122 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.122 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.122 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.122 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.122 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.122 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.122 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:13:33.122 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.122 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.122 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.122 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.122 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.122 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.122 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.123 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:13:33.123 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.810021+0000" 2024-09-22T03:13:33.123 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.123 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.123 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.123 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.123 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.123 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.123 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:13:33.123 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.123 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:13:33.123 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:13:33.123 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.124 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-22T03:13:33.124 INFO:teuthology.orchestra.run.smithi196.stdout: "type": "client_request", 2024-09-22T03:13:33.124 INFO:teuthology.orchestra.run.smithi196.stdout: "id": 393, 2024-09-22T03:13:33.124 INFO:teuthology.orchestra.run.smithi196.stdout: "detail": { 2024-09-22T03:13:33.124 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.124 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:13:33.124 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.792849+0000" 2024-09-22T03:13:33.124 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.124 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.124 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:13:33.124 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.792860+0000" 2024-09-22T03:13:33.124 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.124 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.124 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.124 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.125 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.125 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.125 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:13:33.125 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.792944+0000" 2024-09-22T03:13:33.125 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.125 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.125 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.125 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.125 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.125 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.125 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:13:33.125 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.125 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.125 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.125 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.126 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.126 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.126 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.126 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:13:33.126 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.793002+0000" 2024-09-22T03:13:33.126 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.126 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.126 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.126 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.126 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.126 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.126 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:13:33.126 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.793056+0000" 2024-09-22T03:13:33.127 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.127 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.127 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.127 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.127 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.127 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.127 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:13:33.127 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.127 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.127 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.127 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.127 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.127 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.128 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.128 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:13:33.128 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.821123+0000" 2024-09-22T03:13:33.128 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.128 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.128 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:13:33.128 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.793249+0000" 2024-09-22T03:13:33.128 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.128 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.128 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.128 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.128 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.128 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.128 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:13:33.128 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.129 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.129 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.129 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.129 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.129 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.129 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.129 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:13:33.129 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.793348+0000" 2024-09-22T03:13:33.129 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.129 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.129 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.129 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.129 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.129 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.130 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:13:33.130 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.130 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.130 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.130 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.130 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.130 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.130 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.130 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:13:33.130 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.793441+0000" 2024-09-22T03:13:33.130 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.130 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.130 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.131 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.131 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.131 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.131 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:13:33.131 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.131 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.131 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.131 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.131 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.131 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.131 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.131 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:13:33.131 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.131 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.132 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.132 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.132 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.132 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.132 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.132 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:13:33.132 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.132 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.132 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.132 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.132 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.132 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.132 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.132 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:13:33.132 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.794112+0000" 2024-09-22T03:13:33.133 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.133 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.133 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.133 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.133 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.133 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.133 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:13:33.133 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.133 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.133 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.133 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.133 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.133 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.133 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.133 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:13:33.134 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.795954+0000" 2024-09-22T03:13:33.134 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.134 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.134 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.134 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.134 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.134 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.134 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:13:33.134 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.134 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.134 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.134 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.134 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.134 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.135 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.135 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:13:33.135 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.820777+0000" 2024-09-22T03:13:33.135 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.135 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.135 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.135 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.135 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.135 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.135 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:13:33.135 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.135 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:13:33.135 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:13:33.135 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.136 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-22T03:13:33.136 INFO:teuthology.orchestra.run.smithi196.stdout: "type": "client_request", 2024-09-22T03:13:33.136 INFO:teuthology.orchestra.run.smithi196.stdout: "id": 395, 2024-09-22T03:13:33.136 INFO:teuthology.orchestra.run.smithi196.stdout: "detail": { 2024-09-22T03:13:33.136 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.136 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:13:33.136 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.801806+0000" 2024-09-22T03:13:33.136 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.136 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.136 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:13:33.136 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.801811+0000" 2024-09-22T03:13:33.136 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.136 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.137 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.137 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.137 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.137 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.137 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:13:33.137 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.802014+0000" 2024-09-22T03:13:33.137 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.137 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.137 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.137 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.137 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.137 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.137 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:13:33.137 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.138 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.138 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.138 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.138 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.138 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.138 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.138 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:13:33.138 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.802128+0000" 2024-09-22T03:13:33.138 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.138 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.138 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.138 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.138 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.139 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.139 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:13:33.139 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.802258+0000" 2024-09-22T03:13:33.139 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.139 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.139 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.139 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.139 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.139 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.139 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:13:33.139 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.139 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.139 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.139 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.140 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.140 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.140 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.140 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:13:33.140 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.832861+0000" 2024-09-22T03:13:33.140 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.140 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.140 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:13:33.140 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.802538+0000" 2024-09-22T03:13:33.140 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.140 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.140 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.140 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.140 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.140 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.141 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:13:33.141 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.141 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.141 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.141 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.141 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.141 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.141 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.141 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:13:33.141 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.802646+0000" 2024-09-22T03:13:33.142 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.142 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.142 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.142 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.142 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.142 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.142 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:13:33.142 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.142 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.142 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.142 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.142 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.142 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.143 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.143 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:13:33.143 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.802768+0000" 2024-09-22T03:13:33.143 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.143 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.143 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.143 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.143 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.143 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.143 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:13:33.143 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.143 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.143 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.144 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.144 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.144 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.144 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.144 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:13:33.144 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.144 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.144 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.144 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.144 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.144 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.144 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.144 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:13:33.144 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.144 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.145 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.145 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.145 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.145 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.145 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.145 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:13:33.145 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.802947+0000" 2024-09-22T03:13:33.145 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.145 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.145 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.145 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.145 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.145 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.145 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:13:33.146 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.146 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.146 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.146 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.146 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.146 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.146 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.146 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:13:33.146 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.804579+0000" 2024-09-22T03:13:33.146 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.146 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.146 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.146 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.146 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.147 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.147 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:13:33.147 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.147 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.147 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.147 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.147 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.147 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.147 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.147 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:13:33.147 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.832276+0000" 2024-09-22T03:13:33.147 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.147 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.147 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.148 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.148 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.148 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.148 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:13:33.148 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.148 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:13:33.148 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:13:33.148 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.148 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-22T03:13:33.148 INFO:teuthology.orchestra.run.smithi196.stdout: "type": "client_request", 2024-09-22T03:13:33.148 INFO:teuthology.orchestra.run.smithi196.stdout: "id": 401, 2024-09-22T03:13:33.148 INFO:teuthology.orchestra.run.smithi196.stdout: "detail": { 2024-09-22T03:13:33.148 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.148 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:13:33.149 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.836630+0000" 2024-09-22T03:13:33.149 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.149 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.149 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:13:33.149 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.836636+0000" 2024-09-22T03:13:33.149 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.149 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.149 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.149 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.149 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.149 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.149 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:13:33.149 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.836720+0000" 2024-09-22T03:13:33.149 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.149 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.150 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.150 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.150 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.150 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.150 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:13:33.150 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.150 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.150 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.150 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.150 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.150 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.150 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.150 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:13:33.151 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.836771+0000" 2024-09-22T03:13:33.151 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.151 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.151 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.151 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.151 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.151 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.151 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:13:33.151 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.836830+0000" 2024-09-22T03:13:33.151 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.151 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.151 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.152 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.152 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.152 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.152 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:13:33.152 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.152 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.152 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.152 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.152 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.152 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.152 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.152 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:13:33.152 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.853936+0000" 2024-09-22T03:13:33.152 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.153 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.153 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:13:33.153 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.837049+0000" 2024-09-22T03:13:33.153 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.153 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.153 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.153 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.153 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.153 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.153 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:13:33.153 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.153 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.153 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.153 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.154 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.154 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.154 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.154 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:13:33.154 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.837155+0000" 2024-09-22T03:13:33.154 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.154 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.154 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.154 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.154 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.155 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.155 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:13:33.155 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.155 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.155 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.155 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.155 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.155 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.155 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.155 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:13:33.155 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.837241+0000" 2024-09-22T03:13:33.155 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.155 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.155 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.156 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.156 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.156 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.156 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:13:33.156 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.156 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.156 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.156 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.156 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.156 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.156 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.156 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:13:33.156 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.156 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.156 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.157 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.157 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.157 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.157 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.157 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:13:33.157 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.157 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.157 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.157 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.157 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.157 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.157 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.157 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:13:33.157 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.837396+0000" 2024-09-22T03:13:33.158 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.158 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.158 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.158 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.158 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.158 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.158 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:13:33.158 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.158 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.158 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.158 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.158 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.158 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.159 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.159 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:13:33.159 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.839265+0000" 2024-09-22T03:13:33.159 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.159 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.159 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.159 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.159 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.159 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.159 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:13:33.159 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.159 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.159 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.159 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.160 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.160 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.160 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.160 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:13:33.160 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.853047+0000" 2024-09-22T03:13:33.160 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.160 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.160 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.160 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.160 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.160 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.160 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:13:33.160 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.161 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:13:33.161 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:13:33.161 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.161 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-22T03:13:33.161 INFO:teuthology.orchestra.run.smithi196.stdout: "type": "client_request", 2024-09-22T03:13:33.161 INFO:teuthology.orchestra.run.smithi196.stdout: "id": 413, 2024-09-22T03:13:33.161 INFO:teuthology.orchestra.run.smithi196.stdout: "detail": { 2024-09-22T03:13:33.161 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.161 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:13:33.161 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.875862+0000" 2024-09-22T03:13:33.162 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.162 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.162 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:13:33.162 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.875868+0000" 2024-09-22T03:13:33.162 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.162 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.162 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.162 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.162 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.162 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.162 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:13:33.162 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.875998+0000" 2024-09-22T03:13:33.163 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.163 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.163 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.163 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.163 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.163 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.163 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:13:33.163 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.163 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.163 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.163 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.163 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.163 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.163 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.164 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:13:33.164 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.876053+0000" 2024-09-22T03:13:33.164 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.164 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.164 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.164 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.164 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.164 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.164 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:13:33.164 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.876132+0000" 2024-09-22T03:13:33.164 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.164 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.164 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.164 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.164 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.165 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.165 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:13:33.165 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.165 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.165 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.165 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.165 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.165 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.165 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.165 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:13:33.165 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.892763+0000" 2024-09-22T03:13:33.165 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.165 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.165 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:13:33.166 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.876501+0000" 2024-09-22T03:13:33.166 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.166 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.166 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.166 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.166 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.166 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.166 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:13:33.166 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.166 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.166 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.166 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.166 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.167 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.167 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.167 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:13:33.167 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.876760+0000" 2024-09-22T03:13:33.167 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.167 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.167 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.167 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.167 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.167 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.167 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:13:33.167 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.167 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.167 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.167 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.168 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.168 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.168 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.168 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:13:33.168 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.876923+0000" 2024-09-22T03:13:33.168 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.168 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.169 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.169 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.169 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.169 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.169 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:13:33.169 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.169 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.169 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.169 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.169 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.169 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.169 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.169 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:13:33.169 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.170 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.170 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.170 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.170 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.170 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.170 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.170 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:13:33.171 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.171 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.171 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.171 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.171 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.171 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.171 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.171 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:13:33.171 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.877159+0000" 2024-09-22T03:13:33.172 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.172 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.172 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.172 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.172 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.172 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.172 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:13:33.172 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.172 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.172 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.172 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.172 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.172 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.172 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.172 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:13:33.173 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.879038+0000" 2024-09-22T03:13:33.173 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.173 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.173 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.173 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.173 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.173 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.173 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:13:33.173 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.173 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.173 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.173 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.173 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.173 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.173 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.174 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:13:33.174 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.892413+0000" 2024-09-22T03:13:33.174 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.174 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.174 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.174 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.174 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.174 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.174 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:13:33.174 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.175 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:13:33.175 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:13:33.175 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.175 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-22T03:13:33.175 INFO:teuthology.orchestra.run.smithi196.stdout: "type": "client_request", 2024-09-22T03:13:33.175 INFO:teuthology.orchestra.run.smithi196.stdout: "id": 411, 2024-09-22T03:13:33.175 INFO:teuthology.orchestra.run.smithi196.stdout: "detail": { 2024-09-22T03:13:33.175 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.175 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:13:33.175 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.866532+0000" 2024-09-22T03:13:33.175 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.175 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.175 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:13:33.175 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.866543+0000" 2024-09-22T03:13:33.175 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.176 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.176 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.176 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.176 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.176 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.176 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:13:33.176 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.866668+0000" 2024-09-22T03:13:33.176 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.176 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.176 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.176 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.176 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.176 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.176 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:13:33.177 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.177 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.177 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.177 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.177 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.177 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.177 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.177 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:13:33.177 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.866750+0000" 2024-09-22T03:13:33.177 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.177 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.177 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.177 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.177 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.177 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.178 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:13:33.178 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.866844+0000" 2024-09-22T03:13:33.178 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.178 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.178 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.178 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.178 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.178 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.179 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:13:33.179 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.179 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.179 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.179 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.179 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.179 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.179 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.179 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:13:33.180 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.900177+0000" 2024-09-22T03:13:33.180 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.180 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.180 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:13:33.180 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.867178+0000" 2024-09-22T03:13:33.180 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.180 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.180 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.180 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.180 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.180 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.180 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:13:33.180 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.180 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.181 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.181 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.181 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.181 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.181 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.181 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:13:33.181 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.867369+0000" 2024-09-22T03:13:33.181 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.181 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.181 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.181 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.181 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.181 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.181 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:13:33.181 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.182 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.182 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.182 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.182 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.182 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.182 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.182 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:13:33.182 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.867541+0000" 2024-09-22T03:13:33.182 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.182 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.183 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.183 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.183 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.183 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.183 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:13:33.183 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.183 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.183 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.183 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.183 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.183 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.183 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.183 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:13:33.183 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.184 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.184 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.184 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.184 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.184 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.184 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.184 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:13:33.184 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.184 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.184 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.184 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.184 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.184 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.184 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.184 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:13:33.185 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.867779+0000" 2024-09-22T03:13:33.185 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.185 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.185 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.185 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.185 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.185 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.185 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:13:33.185 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.185 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.185 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.185 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.185 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.185 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.186 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.186 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:13:33.186 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.869811+0000" 2024-09-22T03:13:33.186 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.186 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.186 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.186 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.186 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.186 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.186 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:13:33.186 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.186 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.186 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.187 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.187 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.187 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.187 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.187 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:13:33.187 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.899318+0000" 2024-09-22T03:13:33.187 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.187 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.188 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.188 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.188 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.188 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.188 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:13:33.188 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.188 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:13:33.188 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:13:33.188 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.188 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-22T03:13:33.188 INFO:teuthology.orchestra.run.smithi196.stdout: "type": "client_request", 2024-09-22T03:13:33.188 INFO:teuthology.orchestra.run.smithi196.stdout: "id": 417, 2024-09-22T03:13:33.188 INFO:teuthology.orchestra.run.smithi196.stdout: "detail": { 2024-09-22T03:13:33.188 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.188 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:13:33.189 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.893456+0000" 2024-09-22T03:13:33.189 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.189 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.189 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:13:33.189 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.893463+0000" 2024-09-22T03:13:33.189 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.189 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.189 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.189 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.189 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.189 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.189 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:13:33.189 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.893547+0000" 2024-09-22T03:13:33.189 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.189 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.190 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.190 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.190 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.190 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.190 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:13:33.190 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.190 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.190 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.190 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.190 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.190 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.190 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.190 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:13:33.191 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.893593+0000" 2024-09-22T03:13:33.191 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.191 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.191 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.191 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.191 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.191 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.191 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:13:33.191 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.893647+0000" 2024-09-22T03:13:33.191 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.191 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.191 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.191 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.191 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.192 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.192 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:13:33.192 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.192 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.192 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.192 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.192 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.192 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.192 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.192 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:13:33.192 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.902030+0000" 2024-09-22T03:13:33.192 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.192 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.193 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:13:33.193 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.893836+0000" 2024-09-22T03:13:33.193 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.193 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.193 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.193 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.193 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.193 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.193 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:13:33.194 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.194 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.194 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.194 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.194 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.194 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.194 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.194 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:13:33.195 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.893932+0000" 2024-09-22T03:13:33.195 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.195 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.195 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.195 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.195 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.195 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.195 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:13:33.195 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.195 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.195 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.195 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.195 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.196 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.196 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.196 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:13:33.196 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.894025+0000" 2024-09-22T03:13:33.196 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.196 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.196 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.196 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.196 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.197 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.197 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:13:33.197 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.197 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.197 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.197 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.197 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.197 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.197 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.197 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:13:33.197 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.197 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.198 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.198 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.198 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.198 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.198 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.198 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:13:33.198 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.198 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.198 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.199 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.199 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.199 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.199 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.199 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:13:33.199 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.894184+0000" 2024-09-22T03:13:33.199 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.199 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.199 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.199 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.199 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.199 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.199 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:13:33.200 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.200 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.200 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.200 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.200 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.200 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.200 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.200 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:13:33.200 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.895346+0000" 2024-09-22T03:13:33.200 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.200 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.200 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.200 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.200 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.201 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.201 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:13:33.201 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.201 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.201 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.201 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.201 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.201 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.201 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.201 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:13:33.201 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.901698+0000" 2024-09-22T03:13:33.201 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.201 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.201 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.201 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.202 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.202 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.202 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:13:33.202 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.202 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:13:33.202 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:13:33.202 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.202 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-22T03:13:33.202 INFO:teuthology.orchestra.run.smithi196.stdout: "type": "client_request", 2024-09-22T03:13:33.202 INFO:teuthology.orchestra.run.smithi196.stdout: "id": 419, 2024-09-22T03:13:33.202 INFO:teuthology.orchestra.run.smithi196.stdout: "detail": { 2024-09-22T03:13:33.202 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.202 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:13:33.202 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.903799+0000" 2024-09-22T03:13:33.203 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.203 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.203 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:13:33.203 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.903803+0000" 2024-09-22T03:13:33.203 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.203 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.203 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.203 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.203 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.203 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.203 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:13:33.203 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.903897+0000" 2024-09-22T03:13:33.203 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.203 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.204 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.204 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.204 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.204 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.204 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:13:33.204 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.204 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.204 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.204 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.204 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.204 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.204 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.204 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:13:33.204 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.903984+0000" 2024-09-22T03:13:33.205 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.205 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.205 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.205 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.205 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.205 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.205 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:13:33.205 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.904031+0000" 2024-09-22T03:13:33.205 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.205 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.205 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.205 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.205 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.205 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.205 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:13:33.206 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.206 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.206 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.206 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.206 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.206 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.207 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.207 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:13:33.207 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.925593+0000" 2024-09-22T03:13:33.207 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.207 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.207 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:13:33.208 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.904230+0000" 2024-09-22T03:13:33.208 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.208 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.208 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.208 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.208 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.209 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.209 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:13:33.209 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.209 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.209 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.209 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.209 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.210 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.210 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.210 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:13:33.210 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.904317+0000" 2024-09-22T03:13:33.210 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.210 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.211 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.211 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.211 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.211 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.211 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:13:33.211 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.212 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.212 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.212 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.212 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.212 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.212 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.212 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:13:33.213 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.904396+0000" 2024-09-22T03:13:33.213 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.213 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.213 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.213 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.213 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.214 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.214 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:13:33.214 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.214 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.214 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.214 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.214 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.214 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.214 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.215 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:13:33.215 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.215 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.215 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.215 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.215 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.215 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.215 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.215 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:13:33.215 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.215 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.215 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.215 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.215 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.216 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.216 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.216 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:13:33.216 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.904519+0000" 2024-09-22T03:13:33.216 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.216 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.216 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.216 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.216 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.216 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.216 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:13:33.216 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.216 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.216 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.217 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.217 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.217 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.217 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.217 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:13:33.217 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.907380+0000" 2024-09-22T03:13:33.217 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.217 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.217 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.217 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.217 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.217 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.217 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:13:33.217 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.217 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.218 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.218 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.218 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.218 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.218 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.218 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:13:33.218 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.925019+0000" 2024-09-22T03:13:33.218 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.218 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.218 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.218 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.218 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.218 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.219 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:13:33.219 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.219 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:13:33.219 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:13:33.219 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.219 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-22T03:13:33.219 INFO:teuthology.orchestra.run.smithi196.stdout: "type": "client_request", 2024-09-22T03:13:33.219 INFO:teuthology.orchestra.run.smithi196.stdout: "id": 421, 2024-09-22T03:13:33.219 INFO:teuthology.orchestra.run.smithi196.stdout: "detail": { 2024-09-22T03:13:33.219 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.219 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:13:33.220 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.916397+0000" 2024-09-22T03:13:33.220 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.220 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.220 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:13:33.220 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.916404+0000" 2024-09-22T03:13:33.220 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.220 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.220 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.220 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.220 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.220 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.220 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:13:33.221 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.916508+0000" 2024-09-22T03:13:33.221 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.221 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.221 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.221 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.221 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.221 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.221 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:13:33.221 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.221 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.221 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.221 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.221 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.221 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.222 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.222 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:13:33.222 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.916606+0000" 2024-09-22T03:13:33.222 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.222 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.222 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.222 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.222 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.222 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.222 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:13:33.222 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.916733+0000" 2024-09-22T03:13:33.222 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.222 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.223 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.223 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.223 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.223 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.223 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:13:33.223 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.223 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.223 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.223 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.223 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.223 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.223 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.223 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:13:33.223 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.934730+0000" 2024-09-22T03:13:33.224 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.224 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.224 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:13:33.224 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.917053+0000" 2024-09-22T03:13:33.224 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.224 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.224 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.224 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.224 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.224 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.224 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:13:33.224 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.224 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.224 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.224 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.224 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.225 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.225 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.225 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:13:33.225 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.917209+0000" 2024-09-22T03:13:33.225 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.225 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.225 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.225 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.225 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.225 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.225 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:13:33.225 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.225 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.225 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.225 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.226 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.226 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.226 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.226 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:13:33.226 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.917677+0000" 2024-09-22T03:13:33.226 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.227 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.227 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.227 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.227 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.227 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.227 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:13:33.227 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.227 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.227 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.227 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.227 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.228 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.228 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.228 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:13:33.228 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.228 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.228 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.228 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.228 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.228 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.228 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.228 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:13:33.228 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.228 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.228 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.229 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.229 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.229 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.229 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.229 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:13:33.229 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.918014+0000" 2024-09-22T03:13:33.229 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.229 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.229 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.229 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.229 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.229 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.229 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:13:33.229 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.229 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.230 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.230 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.230 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.230 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.230 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.230 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:13:33.230 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.920323+0000" 2024-09-22T03:13:33.230 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.230 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.230 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.230 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.230 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.230 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.231 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:13:33.231 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.231 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.231 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.231 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.231 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.231 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.231 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.231 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:13:33.231 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.934361+0000" 2024-09-22T03:13:33.231 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.231 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.231 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.231 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.231 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.232 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.232 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:13:33.232 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.232 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:13:33.232 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:13:33.232 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.232 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-22T03:13:33.232 INFO:teuthology.orchestra.run.smithi196.stdout: "type": "client_request", 2024-09-22T03:13:33.232 INFO:teuthology.orchestra.run.smithi196.stdout: "id": 418, 2024-09-22T03:13:33.232 INFO:teuthology.orchestra.run.smithi196.stdout: "detail": { 2024-09-22T03:13:33.232 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.232 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:13:33.232 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.902328+0000" 2024-09-22T03:13:33.232 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.232 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.233 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:13:33.233 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.902334+0000" 2024-09-22T03:13:33.233 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.233 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.233 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.233 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.233 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.233 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.233 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:13:33.233 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.902434+0000" 2024-09-22T03:13:33.233 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.233 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.233 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.233 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.233 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.234 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.234 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:13:33.234 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.234 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.234 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.234 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.234 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.234 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.234 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.234 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:13:33.234 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.902481+0000" 2024-09-22T03:13:33.234 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.234 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.235 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.235 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.235 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.235 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.235 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:13:33.235 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.902526+0000" 2024-09-22T03:13:33.235 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.235 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.235 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.235 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.236 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.236 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.236 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:13:33.236 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.236 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.236 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.236 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.236 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.236 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.236 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.236 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:13:33.236 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.941818+0000" 2024-09-22T03:13:33.236 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.237 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.237 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:13:33.237 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.902706+0000" 2024-09-22T03:13:33.237 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.237 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.237 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.237 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.237 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.237 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.237 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:13:33.237 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.237 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.237 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.237 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.237 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.238 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.238 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.238 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:13:33.238 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.902823+0000" 2024-09-22T03:13:33.238 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.238 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.238 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.238 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.238 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.238 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.238 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:13:33.238 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.238 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.238 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.239 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.239 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.239 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.239 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.239 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:13:33.239 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.902971+0000" 2024-09-22T03:13:33.239 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.239 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.239 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.239 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.239 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.239 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.239 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:13:33.239 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.239 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.240 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.240 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.240 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.240 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.240 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.240 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:13:33.240 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.240 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.240 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.240 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.240 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.240 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.240 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.240 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:13:33.241 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.241 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.241 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.241 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.241 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.241 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.241 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.241 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:13:33.241 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.903215+0000" 2024-09-22T03:13:33.241 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.241 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.241 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.241 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.241 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.242 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.242 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:13:33.242 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.242 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.242 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.242 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.242 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.242 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.242 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.242 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:13:33.242 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.905177+0000" 2024-09-22T03:13:33.243 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.243 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.243 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.243 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.243 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.243 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.243 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:13:33.243 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.243 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.243 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.243 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.243 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.243 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.243 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.244 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:13:33.244 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.941434+0000" 2024-09-22T03:13:33.244 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.244 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.244 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.244 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.244 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.244 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.244 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:13:33.244 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.244 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:13:33.244 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:13:33.244 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.244 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-22T03:13:33.244 INFO:teuthology.orchestra.run.smithi196.stdout: "type": "client_request", 2024-09-22T03:13:33.245 INFO:teuthology.orchestra.run.smithi196.stdout: "id": 420, 2024-09-22T03:13:33.245 INFO:teuthology.orchestra.run.smithi196.stdout: "detail": { 2024-09-22T03:13:33.245 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.245 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:13:33.245 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.910568+0000" 2024-09-22T03:13:33.245 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.245 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.245 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:13:33.245 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.910579+0000" 2024-09-22T03:13:33.245 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.245 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.245 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.245 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.245 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.246 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.246 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:13:33.246 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.910722+0000" 2024-09-22T03:13:33.246 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.246 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.246 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.246 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.246 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.246 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.246 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:13:33.246 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.246 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.246 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.246 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.247 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.247 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.247 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.247 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:13:33.247 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.911653+0000" 2024-09-22T03:13:33.247 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.247 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.247 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.247 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.247 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.247 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.247 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:13:33.247 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.911716+0000" 2024-09-22T03:13:33.247 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.247 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.248 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.248 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.248 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.248 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.248 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:13:33.248 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.248 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.248 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.248 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.248 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.248 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.248 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.248 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:13:33.248 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.942575+0000" 2024-09-22T03:13:33.249 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.249 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.249 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:13:33.249 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.911948+0000" 2024-09-22T03:13:33.249 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.249 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.249 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.249 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.249 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.249 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.249 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:13:33.249 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.249 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.249 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.249 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.250 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.250 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.250 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.250 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:13:33.250 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.912071+0000" 2024-09-22T03:13:33.250 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.250 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.250 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.250 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.250 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.250 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.250 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:13:33.250 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.250 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.250 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.251 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.251 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.251 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.251 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.251 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:13:33.251 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.912157+0000" 2024-09-22T03:13:33.251 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.251 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.251 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.251 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.251 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.251 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.252 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:13:33.252 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.252 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.252 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.252 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.252 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.252 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.252 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.252 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:13:33.252 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.252 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.252 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.252 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.252 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.252 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.253 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.253 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:13:33.253 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.253 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.253 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.253 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.253 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.253 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.253 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.253 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:13:33.253 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.912294+0000" 2024-09-22T03:13:33.253 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.253 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.253 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.253 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.254 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.254 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.254 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:13:33.254 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.254 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.254 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.254 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.254 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.254 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.254 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.254 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:13:33.254 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.914376+0000" 2024-09-22T03:13:33.254 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.254 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.255 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.255 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.255 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.255 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.255 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:13:33.255 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.255 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.255 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.255 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.255 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.255 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.255 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.255 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:13:33.255 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.942304+0000" 2024-09-22T03:13:33.256 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.256 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.256 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.256 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.256 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.256 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.256 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:13:33.256 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.256 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:13:33.256 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:13:33.256 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.256 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-22T03:13:33.256 INFO:teuthology.orchestra.run.smithi196.stdout: "type": "client_request", 2024-09-22T03:13:33.256 INFO:teuthology.orchestra.run.smithi196.stdout: "id": 427, 2024-09-22T03:13:33.257 INFO:teuthology.orchestra.run.smithi196.stdout: "detail": { 2024-09-22T03:13:33.257 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.257 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:13:33.257 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.942839+0000" 2024-09-22T03:13:33.257 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.257 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.257 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:13:33.257 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.942846+0000" 2024-09-22T03:13:33.257 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.257 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.257 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.257 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.257 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.257 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.258 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:13:33.258 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.942989+0000" 2024-09-22T03:13:33.258 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.258 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.258 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.258 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.258 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.258 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.258 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:13:33.258 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.258 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.258 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.258 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.258 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.259 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.259 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.259 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:13:33.259 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.943067+0000" 2024-09-22T03:13:33.259 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.259 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.259 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.259 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.259 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.259 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.259 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:13:33.259 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.943324+0000" 2024-09-22T03:13:33.259 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.259 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.260 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.260 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.260 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.260 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.260 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:13:33.260 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.260 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.260 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.260 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.260 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.260 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.260 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.260 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:13:33.260 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.261 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.261 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.261 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:13:33.261 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.943558+0000" 2024-09-22T03:13:33.261 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.261 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.261 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.261 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.261 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.261 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.261 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:13:33.261 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.261 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.261 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.261 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.262 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.262 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.262 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.262 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:13:33.262 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.943660+0000" 2024-09-22T03:13:33.262 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.262 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.262 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.262 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.262 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.262 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.262 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:13:33.262 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.263 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.263 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.263 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.263 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.263 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.263 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.263 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:13:33.263 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.943749+0000" 2024-09-22T03:13:33.263 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.263 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.263 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.263 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.263 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.263 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.263 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:13:33.264 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.264 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.264 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.264 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.264 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.264 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.264 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.264 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:13:33.264 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.264 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.264 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.264 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.264 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.264 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.265 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.265 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:13:33.265 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.265 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.265 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.265 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.265 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.265 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.265 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.265 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:13:33.265 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.943908+0000" 2024-09-22T03:13:33.265 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.265 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.265 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.265 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.266 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.266 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.266 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:13:33.266 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.266 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.266 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.266 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.266 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.266 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.266 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.266 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:13:33.266 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.945259+0000" 2024-09-22T03:13:33.266 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.266 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.267 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.267 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.267 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.267 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.267 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:13:33.267 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.267 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.267 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.267 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.267 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.267 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.267 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.267 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:13:33.267 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.267 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.268 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.268 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.268 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.268 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.268 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.268 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:13:33.268 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.268 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:13:33.268 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:13:33.268 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.268 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-22T03:13:33.268 INFO:teuthology.orchestra.run.smithi196.stdout: "type": "client_request", 2024-09-22T03:13:33.268 INFO:teuthology.orchestra.run.smithi196.stdout: "id": 429, 2024-09-22T03:13:33.268 INFO:teuthology.orchestra.run.smithi196.stdout: "detail": { 2024-09-22T03:13:33.269 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.269 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:13:33.269 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.953578+0000" 2024-09-22T03:13:33.269 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.269 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.269 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:13:33.269 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.953583+0000" 2024-09-22T03:13:33.269 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.269 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.269 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.269 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.269 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.269 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.269 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:13:33.269 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.953666+0000" 2024-09-22T03:13:33.270 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.270 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.270 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.270 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.270 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.270 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.270 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:13:33.270 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.271 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.271 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.271 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.271 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.271 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.271 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.271 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:13:33.271 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.953715+0000" 2024-09-22T03:13:33.271 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.271 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.271 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.271 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.271 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.271 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.271 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:13:33.272 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.953904+0000" 2024-09-22T03:13:33.272 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.272 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.272 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.272 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.272 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.272 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.272 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:13:33.272 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.272 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.272 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.272 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.272 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.272 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.273 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.273 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:13:33.273 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.273 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.273 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.273 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:13:33.273 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.954273+0000" 2024-09-22T03:13:33.273 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.273 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.273 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.273 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.273 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.273 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.273 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:13:33.273 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.274 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.274 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.274 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.274 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.274 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.274 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.274 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:13:33.274 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.954464+0000" 2024-09-22T03:13:33.274 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.274 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.274 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.274 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.274 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.274 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.275 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:13:33.275 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.275 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.275 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.275 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.275 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.275 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.275 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.275 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:13:33.275 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.954596+0000" 2024-09-22T03:13:33.275 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.275 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.275 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.276 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.276 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.276 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.276 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:13:33.276 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.276 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.276 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.277 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.277 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.277 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.277 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.277 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:13:33.277 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.277 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.277 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.277 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.277 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.277 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.277 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.278 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:13:33.278 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.278 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.278 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.278 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.278 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.278 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.278 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.278 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:13:33.278 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.954833+0000" 2024-09-22T03:13:33.278 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.278 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.278 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.279 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.279 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.279 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.279 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:13:33.279 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.279 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.279 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.279 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.279 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.279 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.279 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.279 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:13:33.279 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.956255+0000" 2024-09-22T03:13:33.280 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.280 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.280 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.280 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.280 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.280 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.280 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:13:33.280 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.280 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.280 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.280 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.280 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.280 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.281 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.281 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:13:33.281 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.281 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.281 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.281 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.281 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.281 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.281 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.281 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:13:33.281 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.281 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:13:33.281 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:13:33.281 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.282 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-22T03:13:33.282 INFO:teuthology.orchestra.run.smithi196.stdout: "type": "client_request", 2024-09-22T03:13:33.282 INFO:teuthology.orchestra.run.smithi196.stdout: "id": 434, 2024-09-22T03:13:33.282 INFO:teuthology.orchestra.run.smithi196.stdout: "detail": { 2024-09-22T03:13:33.282 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.282 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:13:33.282 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.975958+0000" 2024-09-22T03:13:33.282 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.282 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.282 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:13:33.282 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.975962+0000" 2024-09-22T03:13:33.282 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.282 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.283 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.283 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.283 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.283 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.283 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:13:33.283 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.976037+0000" 2024-09-22T03:13:33.283 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.283 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.283 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.283 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.283 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.283 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.284 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:13:33.284 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.284 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.284 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.284 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.284 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.284 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.284 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.284 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:13:33.284 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.976086+0000" 2024-09-22T03:13:33.284 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.284 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.284 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.284 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.285 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.285 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.285 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:13:33.285 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.978198+0000" 2024-09-22T03:13:33.285 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.285 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.285 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.285 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.285 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.285 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.285 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:13:33.285 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.285 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.285 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.286 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.286 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.286 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.286 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.286 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:13:33.286 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.286 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.286 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.286 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:13:33.286 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.978543+0000" 2024-09-22T03:13:33.286 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.286 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.286 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.287 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.287 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.287 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.287 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:13:33.287 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.287 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.287 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.287 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.287 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.287 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.287 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.287 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:13:33.287 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.978722+0000" 2024-09-22T03:13:33.288 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.288 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.288 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.288 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.288 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.288 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.288 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:13:33.288 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.288 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.288 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.288 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.288 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.288 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.288 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.289 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:13:33.289 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.978869+0000" 2024-09-22T03:13:33.289 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.289 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.289 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.289 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.289 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.289 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.289 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:13:33.289 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.289 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.289 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.289 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.289 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.290 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.290 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.290 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:13:33.290 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.290 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.290 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.290 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.290 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.290 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.290 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.290 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:13:33.290 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.290 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.291 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.291 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.291 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.291 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.291 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.291 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:13:33.291 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.979127+0000" 2024-09-22T03:13:33.291 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.291 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.291 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.291 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.291 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.291 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.291 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:13:33.292 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.292 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.292 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.292 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.292 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.292 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.292 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.292 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:13:33.292 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.980548+0000" 2024-09-22T03:13:33.292 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.292 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.292 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.292 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.292 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.293 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.293 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:13:33.293 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.293 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.293 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.293 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.293 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.293 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.293 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.293 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:13:33.293 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.293 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.293 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.293 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.294 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.294 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.294 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.294 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:13:33.294 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.294 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:13:33.294 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:13:33.294 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.294 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-22T03:13:33.294 INFO:teuthology.orchestra.run.smithi196.stdout: "type": "client_request", 2024-09-22T03:13:33.294 INFO:teuthology.orchestra.run.smithi196.stdout: "id": 309, 2024-09-22T03:13:33.294 INFO:teuthology.orchestra.run.smithi196.stdout: "detail": { 2024-09-22T03:13:33.295 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.295 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:13:33.295 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.510981+0000" 2024-09-22T03:13:33.295 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.295 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.295 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:13:33.295 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.510989+0000" 2024-09-22T03:13:33.295 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.295 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.295 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.295 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.295 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.295 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.296 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:13:33.296 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.511578+0000" 2024-09-22T03:13:33.296 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.296 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.296 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.296 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.296 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.296 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.296 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:13:33.296 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.296 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.296 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.296 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.296 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.296 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.297 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.297 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:13:33.297 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.511695+0000" 2024-09-22T03:13:33.297 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.297 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.297 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.297 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.297 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.297 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.297 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:13:33.297 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.512225+0000" 2024-09-22T03:13:33.297 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.297 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.298 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.298 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.298 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.298 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.298 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:13:33.298 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.298 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.298 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.298 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.298 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.298 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.298 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.298 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:13:33.299 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.560023+0000" 2024-09-22T03:13:33.299 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.299 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.299 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:13:33.299 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.512593+0000" 2024-09-22T03:13:33.299 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.299 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.299 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.299 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.299 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.299 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.299 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:13:33.300 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.300 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.300 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.300 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.300 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.300 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.300 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.300 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:13:33.300 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.512771+0000" 2024-09-22T03:13:33.300 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.300 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.300 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.300 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.300 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.300 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.301 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:13:33.301 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.301 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.301 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.301 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.301 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.301 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.301 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.301 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:13:33.301 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.512927+0000" 2024-09-22T03:13:33.301 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.301 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.301 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.301 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.302 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.302 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.302 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:13:33.302 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.302 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.302 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.302 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.302 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.302 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.302 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.302 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:13:33.302 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.302 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.302 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.303 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.303 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.303 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.303 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.303 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:13:33.303 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.303 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.303 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.303 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.303 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.303 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.303 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.303 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:13:33.303 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.513175+0000" 2024-09-22T03:13:33.303 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.304 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.304 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.304 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.304 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.304 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.304 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:13:33.304 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.304 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.304 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.304 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.304 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.304 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.304 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.304 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:13:33.305 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.514551+0000" 2024-09-22T03:13:33.305 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.305 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.305 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.305 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.305 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.305 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.305 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:13:33.305 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.305 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.305 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.305 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.305 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.305 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.305 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.306 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:13:33.306 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.559240+0000" 2024-09-22T03:13:33.306 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.306 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.306 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.306 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.306 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.306 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.306 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:13:33.306 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.306 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:13:33.306 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:13:33.306 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.307 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-22T03:13:33.307 INFO:teuthology.orchestra.run.smithi196.stdout: "type": "client_request", 2024-09-22T03:13:33.307 INFO:teuthology.orchestra.run.smithi196.stdout: "id": 314, 2024-09-22T03:13:33.307 INFO:teuthology.orchestra.run.smithi196.stdout: "detail": { 2024-09-22T03:13:33.307 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.307 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:13:33.307 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.529165+0000" 2024-09-22T03:13:33.307 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.307 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.307 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:13:33.307 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.529172+0000" 2024-09-22T03:13:33.307 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.307 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.307 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.307 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.308 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.308 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.308 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:13:33.308 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.529284+0000" 2024-09-22T03:13:33.308 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.308 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.308 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.308 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.308 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.308 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.308 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:13:33.308 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.308 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.308 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.308 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.309 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.309 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.309 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.309 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:13:33.309 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.530315+0000" 2024-09-22T03:13:33.309 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.309 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.309 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.309 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.309 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.309 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.310 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:13:33.310 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.531578+0000" 2024-09-22T03:13:33.310 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.310 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.310 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.310 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.310 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.310 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.310 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:13:33.310 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.310 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.310 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.310 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.310 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.311 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.311 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.311 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:13:33.311 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.578005+0000" 2024-09-22T03:13:33.311 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.311 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.311 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:13:33.311 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.531845+0000" 2024-09-22T03:13:33.311 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.311 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.311 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.311 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.311 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.312 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.312 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:13:33.312 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.312 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.312 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.312 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.312 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.312 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.312 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.312 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:13:33.312 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.531940+0000" 2024-09-22T03:13:33.312 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.312 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.312 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.313 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.313 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.313 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.313 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:13:33.313 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.313 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.313 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.313 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.313 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.313 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.313 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.313 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:13:33.313 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.532045+0000" 2024-09-22T03:13:33.313 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.314 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.314 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.314 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.314 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.314 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.314 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:13:33.314 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.314 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.314 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.314 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.314 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.314 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.314 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.315 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:13:33.315 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.315 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.315 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.315 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.315 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.315 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.315 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.315 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:13:33.315 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.315 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.315 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.315 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.315 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.316 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.316 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.316 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:13:33.316 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.532316+0000" 2024-09-22T03:13:33.316 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.316 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.316 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.316 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.316 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.316 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.316 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:13:33.316 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.316 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.316 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.316 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.317 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.317 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.317 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.317 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:13:33.317 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.533827+0000" 2024-09-22T03:13:33.317 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.317 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.317 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.317 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.317 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.317 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.317 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:13:33.317 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.317 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.317 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.317 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.318 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.318 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.318 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.318 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:13:33.318 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.577714+0000" 2024-09-22T03:13:33.318 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.318 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.318 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.318 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.318 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.318 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.318 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:13:33.318 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.318 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:13:33.319 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:13:33.319 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.319 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-22T03:13:33.319 INFO:teuthology.orchestra.run.smithi196.stdout: "type": "client_request", 2024-09-22T03:13:33.319 INFO:teuthology.orchestra.run.smithi196.stdout: "id": 320, 2024-09-22T03:13:33.319 INFO:teuthology.orchestra.run.smithi196.stdout: "detail": { 2024-09-22T03:13:33.319 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.319 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:13:33.319 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.550033+0000" 2024-09-22T03:13:33.319 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.319 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.319 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:13:33.319 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.550039+0000" 2024-09-22T03:13:33.319 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.319 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.319 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.320 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.320 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.320 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.320 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:13:33.320 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.550524+0000" 2024-09-22T03:13:33.320 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.320 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.320 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.320 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.320 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.320 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.320 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:13:33.320 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.320 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.320 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.321 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.321 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.321 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.321 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.321 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:13:33.321 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.554173+0000" 2024-09-22T03:13:33.321 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.321 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.321 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.321 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.321 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.321 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.321 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:13:33.321 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.558566+0000" 2024-09-22T03:13:33.321 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.322 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.322 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.322 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.322 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.322 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.322 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:13:33.322 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.322 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.322 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.322 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.322 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.322 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.322 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.323 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:13:33.323 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.600141+0000" 2024-09-22T03:13:33.323 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.323 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.323 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:13:33.323 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.559101+0000" 2024-09-22T03:13:33.323 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.323 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.323 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.323 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.323 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.323 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.323 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:13:33.323 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.324 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.324 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.324 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.324 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.324 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.324 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.324 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:13:33.324 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.559500+0000" 2024-09-22T03:13:33.324 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.324 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.324 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.324 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.324 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.324 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.325 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:13:33.325 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.325 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.325 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.325 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.325 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.325 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.325 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.325 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:13:33.325 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.559810+0000" 2024-09-22T03:13:33.325 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.325 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.325 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.325 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.325 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.326 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.326 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:13:33.326 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.326 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.326 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.326 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.326 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.326 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.326 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.326 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:13:33.326 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.326 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.326 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.326 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.327 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.327 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.327 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.327 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:13:33.327 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.327 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.327 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.327 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.327 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.327 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.327 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.327 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:13:33.327 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.560231+0000" 2024-09-22T03:13:33.327 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.327 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.328 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.328 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.328 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.328 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.328 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:13:33.328 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.328 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.328 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.328 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.328 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.328 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.328 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.328 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:13:33.329 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.561655+0000" 2024-09-22T03:13:33.329 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.329 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.329 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.329 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.329 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.329 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.329 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:13:33.329 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.329 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.329 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.329 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.329 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.329 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.329 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.330 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:13:33.330 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.599716+0000" 2024-09-22T03:13:33.330 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.330 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.330 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.330 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.330 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.330 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.330 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:13:33.330 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.330 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:13:33.330 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:13:33.330 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.330 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-22T03:13:33.331 INFO:teuthology.orchestra.run.smithi196.stdout: "type": "client_request", 2024-09-22T03:13:33.331 INFO:teuthology.orchestra.run.smithi196.stdout: "id": 321, 2024-09-22T03:13:33.331 INFO:teuthology.orchestra.run.smithi196.stdout: "detail": { 2024-09-22T03:13:33.331 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.331 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:13:33.331 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.559890+0000" 2024-09-22T03:13:33.331 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.331 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.331 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:13:33.331 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.559900+0000" 2024-09-22T03:13:33.331 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.331 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.331 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.332 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.332 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.332 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.332 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:13:33.332 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.560899+0000" 2024-09-22T03:13:33.332 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.332 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.332 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.332 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.332 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.332 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.332 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:13:33.332 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.332 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.332 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.333 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.333 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.333 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.333 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.333 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:13:33.333 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.561767+0000" 2024-09-22T03:13:33.333 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.333 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.333 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.333 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.333 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.333 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.333 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:13:33.333 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.563854+0000" 2024-09-22T03:13:33.334 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.334 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.334 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.334 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.334 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.334 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.334 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:13:33.334 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.334 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.334 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.334 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.334 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.334 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.335 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.335 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:13:33.335 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.604660+0000" 2024-09-22T03:13:33.335 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.335 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.335 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:13:33.335 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.564176+0000" 2024-09-22T03:13:33.335 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.335 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.335 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.335 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.335 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.335 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.335 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:13:33.335 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.336 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.336 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.336 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.336 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.336 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.336 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.336 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:13:33.336 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.564273+0000" 2024-09-22T03:13:33.336 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.336 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.336 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.336 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.336 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.337 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.337 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:13:33.337 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.337 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.337 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.337 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.337 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.337 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.337 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.337 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:13:33.337 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.565523+0000" 2024-09-22T03:13:33.337 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.337 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.337 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.337 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.338 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.338 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.338 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:13:33.338 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.338 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.338 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.338 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.338 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.338 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.338 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.338 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:13:33.338 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.338 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.339 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.339 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.339 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.339 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.339 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.339 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:13:33.339 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.339 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.339 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.339 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.339 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.339 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.339 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.339 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:13:33.339 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.565682+0000" 2024-09-22T03:13:33.340 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.340 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.340 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.340 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.340 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.340 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.340 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:13:33.340 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.340 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.340 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.340 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.340 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.340 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.340 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.341 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:13:33.341 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.567181+0000" 2024-09-22T03:13:33.341 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.341 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.341 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.341 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.341 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.341 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.341 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:13:33.341 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.341 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.341 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.341 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.341 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.341 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.342 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.342 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:13:33.342 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.604207+0000" 2024-09-22T03:13:33.342 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.342 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.342 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.342 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.342 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.342 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.342 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:13:33.342 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.343 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:13:33.343 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:13:33.343 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.343 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-22T03:13:33.343 INFO:teuthology.orchestra.run.smithi196.stdout: "type": "client_request", 2024-09-22T03:13:33.343 INFO:teuthology.orchestra.run.smithi196.stdout: "id": 324, 2024-09-22T03:13:33.343 INFO:teuthology.orchestra.run.smithi196.stdout: "detail": { 2024-09-22T03:13:33.343 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.343 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:13:33.343 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.566753+0000" 2024-09-22T03:13:33.343 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.343 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.343 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:13:33.343 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.566762+0000" 2024-09-22T03:13:33.344 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.344 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.344 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.344 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.344 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.344 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.344 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:13:33.344 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.566948+0000" 2024-09-22T03:13:33.344 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.344 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.344 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.344 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.344 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.344 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.345 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:13:33.345 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.345 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.345 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.345 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.345 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.345 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.345 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.345 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:13:33.345 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.567011+0000" 2024-09-22T03:13:33.345 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.345 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.345 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.345 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.346 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.346 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.346 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:13:33.346 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.569942+0000" 2024-09-22T03:13:33.346 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.346 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.346 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.346 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.346 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.346 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.346 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:13:33.346 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.346 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.347 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.347 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.347 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.347 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.347 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.347 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:13:33.347 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.609825+0000" 2024-09-22T03:13:33.348 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.348 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.348 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:13:33.348 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.570223+0000" 2024-09-22T03:13:33.348 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.348 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.348 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.348 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.348 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.349 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.349 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:13:33.349 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.349 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.349 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.349 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.349 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.349 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.349 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.349 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:13:33.349 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.570333+0000" 2024-09-22T03:13:33.349 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.349 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.349 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.350 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.350 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.350 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.350 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:13:33.350 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.350 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.350 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.350 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.350 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.350 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.350 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.350 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:13:33.350 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.570469+0000" 2024-09-22T03:13:33.351 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.351 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.351 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.351 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.351 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.351 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.351 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:13:33.351 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.351 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.351 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.351 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.351 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.351 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.352 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.352 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:13:33.352 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.352 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.352 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.352 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.352 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.352 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.352 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.352 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:13:33.352 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.352 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.352 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.353 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.353 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.353 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.353 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.353 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:13:33.353 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.570607+0000" 2024-09-22T03:13:33.353 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.353 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.353 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.353 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.353 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.353 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.353 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:13:33.353 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.354 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.354 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.354 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.354 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.354 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.354 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.354 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:13:33.354 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.574156+0000" 2024-09-22T03:13:33.354 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.354 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.354 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.354 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.354 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.355 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.355 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:13:33.355 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.355 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.355 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.355 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.355 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.355 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.355 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.355 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:13:33.355 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.609424+0000" 2024-09-22T03:13:33.355 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.355 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.356 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.356 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.356 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.356 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.356 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:13:33.356 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.356 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:13:33.356 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:13:33.356 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.356 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-22T03:13:33.356 INFO:teuthology.orchestra.run.smithi196.stdout: "type": "client_request", 2024-09-22T03:13:33.356 INFO:teuthology.orchestra.run.smithi196.stdout: "id": 330, 2024-09-22T03:13:33.356 INFO:teuthology.orchestra.run.smithi196.stdout: "detail": { 2024-09-22T03:13:33.356 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.357 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:13:33.357 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.585239+0000" 2024-09-22T03:13:33.357 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.357 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.357 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:13:33.357 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.585245+0000" 2024-09-22T03:13:33.357 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.357 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.357 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.357 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.357 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.357 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.357 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:13:33.357 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.586879+0000" 2024-09-22T03:13:33.357 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.358 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.358 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.358 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.358 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.358 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.358 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:13:33.358 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.358 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.358 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.358 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.358 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.358 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.358 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.359 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:13:33.359 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.587713+0000" 2024-09-22T03:13:33.359 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.359 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.359 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.359 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.359 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.359 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.359 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:13:33.359 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.590658+0000" 2024-09-22T03:13:33.359 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.359 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.359 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.359 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.359 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.360 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.360 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:13:33.360 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.360 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.360 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.360 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.360 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.360 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.360 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.360 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:13:33.360 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.618899+0000" 2024-09-22T03:13:33.360 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.360 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.360 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:13:33.360 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.591035+0000" 2024-09-22T03:13:33.361 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.361 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.361 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.361 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.361 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.361 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.361 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:13:33.361 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.361 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.361 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.361 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.361 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.361 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.361 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.362 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:13:33.362 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.591216+0000" 2024-09-22T03:13:33.362 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.362 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.362 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.362 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.362 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.362 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.362 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:13:33.362 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.362 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.362 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.362 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.362 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.362 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.363 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.363 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:13:33.363 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.591382+0000" 2024-09-22T03:13:33.363 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.363 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.363 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.363 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.363 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.363 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.363 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:13:33.363 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.363 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.363 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.363 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.364 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.364 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.364 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.364 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:13:33.364 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.364 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.364 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.364 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.364 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.364 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.364 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.364 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:13:33.364 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.364 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.364 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.365 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.365 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.365 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.365 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.365 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:13:33.365 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.591620+0000" 2024-09-22T03:13:33.365 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.365 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.365 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.365 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.365 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.365 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.365 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:13:33.365 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.366 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.366 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.366 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.366 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.366 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.366 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.366 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:13:33.366 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.593059+0000" 2024-09-22T03:13:33.366 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.366 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.366 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.366 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.366 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.366 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.367 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:13:33.367 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.367 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.367 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.367 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.367 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.367 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.367 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.367 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:13:33.367 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.618434+0000" 2024-09-22T03:13:33.367 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.367 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.367 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.367 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.368 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.368 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.368 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:13:33.368 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.368 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:13:33.368 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:13:33.368 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.368 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-22T03:13:33.368 INFO:teuthology.orchestra.run.smithi196.stdout: "type": "client_request", 2024-09-22T03:13:33.368 INFO:teuthology.orchestra.run.smithi196.stdout: "id": 333, 2024-09-22T03:13:33.368 INFO:teuthology.orchestra.run.smithi196.stdout: "detail": { 2024-09-22T03:13:33.368 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.368 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:13:33.368 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.592930+0000" 2024-09-22T03:13:33.369 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.369 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.369 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:13:33.369 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.592940+0000" 2024-09-22T03:13:33.369 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.369 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.369 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.369 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.369 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.369 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.369 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:13:33.369 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.593447+0000" 2024-09-22T03:13:33.369 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.369 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.370 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.370 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.370 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.370 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.370 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:13:33.370 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.370 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.370 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.370 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.370 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.370 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.370 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.370 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:13:33.370 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.593529+0000" 2024-09-22T03:13:33.370 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.371 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.371 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.371 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.371 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.371 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.371 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:13:33.371 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.596024+0000" 2024-09-22T03:13:33.371 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.371 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.371 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.371 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.371 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.371 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.371 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:13:33.372 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.372 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.372 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.372 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.372 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.372 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.372 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.372 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:13:33.372 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.638870+0000" 2024-09-22T03:13:33.372 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.372 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.372 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:13:33.372 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.596588+0000" 2024-09-22T03:13:33.372 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.372 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.373 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.373 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.373 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.373 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.373 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:13:33.373 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.373 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.373 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.373 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.373 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.373 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.373 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.373 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:13:33.373 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.598559+0000" 2024-09-22T03:13:33.373 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.374 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.374 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.374 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.374 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.374 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.374 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:13:33.374 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.374 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.374 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.374 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.374 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.374 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.374 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.374 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:13:33.375 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.598749+0000" 2024-09-22T03:13:33.375 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.375 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.375 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.375 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.375 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.375 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.375 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:13:33.375 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.375 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.375 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.375 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.375 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.375 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.375 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.376 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:13:33.376 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.376 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.376 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.376 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.376 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.376 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.376 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.376 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:13:33.376 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.376 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.376 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.376 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.376 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.377 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.377 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.377 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:13:33.377 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.598965+0000" 2024-09-22T03:13:33.377 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.377 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.377 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.377 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.377 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.377 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.377 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:13:33.377 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.377 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.377 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.378 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.378 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.378 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.378 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.378 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:13:33.378 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.600917+0000" 2024-09-22T03:13:33.378 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.378 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.378 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.378 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.378 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.378 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.378 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:13:33.378 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.378 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.379 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.379 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.379 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.379 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.379 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.379 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:13:33.379 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.636803+0000" 2024-09-22T03:13:33.379 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.379 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.379 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.379 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.380 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.380 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.380 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:13:33.380 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.380 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:13:33.380 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:13:33.380 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.380 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-22T03:13:33.380 INFO:teuthology.orchestra.run.smithi196.stdout: "type": "client_request", 2024-09-22T03:13:33.380 INFO:teuthology.orchestra.run.smithi196.stdout: "id": 336, 2024-09-22T03:13:33.381 INFO:teuthology.orchestra.run.smithi196.stdout: "detail": { 2024-09-22T03:13:33.381 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.381 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:13:33.381 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.609109+0000" 2024-09-22T03:13:33.381 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.381 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.381 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:13:33.381 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.609117+0000" 2024-09-22T03:13:33.381 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.381 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.381 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.381 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.381 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.381 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.382 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:13:33.382 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.609920+0000" 2024-09-22T03:13:33.382 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.382 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.382 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.382 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.382 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.382 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.382 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:13:33.382 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.382 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.382 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.383 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.383 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.383 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.383 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.383 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:13:33.383 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.610064+0000" 2024-09-22T03:13:33.383 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.383 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.383 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.383 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.383 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.383 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.383 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:13:33.384 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.610240+0000" 2024-09-22T03:13:33.384 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.384 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.384 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.384 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.384 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.384 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.384 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:13:33.384 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.384 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.384 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.384 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.385 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.385 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.385 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.385 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:13:33.385 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.647950+0000" 2024-09-22T03:13:33.385 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.385 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.385 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:13:33.385 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.610558+0000" 2024-09-22T03:13:33.385 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.385 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.385 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.385 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.386 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.386 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.386 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:13:33.386 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.386 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.386 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.386 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.386 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.386 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.386 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.386 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:13:33.386 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.610736+0000" 2024-09-22T03:13:33.386 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.386 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.386 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.387 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.387 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.387 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.387 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:13:33.387 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.387 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.387 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.387 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.387 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.387 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.387 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.387 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:13:33.387 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.610896+0000" 2024-09-22T03:13:33.387 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.388 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.388 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.388 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.388 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.388 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.388 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:13:33.388 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.388 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.388 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.388 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.388 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.388 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.388 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.388 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:13:33.389 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.389 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.389 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.389 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.389 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.389 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.389 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.389 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:13:33.389 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.389 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.389 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.389 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.389 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.389 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.390 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.390 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:13:33.390 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.611146+0000" 2024-09-22T03:13:33.390 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.390 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.390 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.390 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.390 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.390 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.390 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:13:33.390 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.390 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.390 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.390 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.391 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.391 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.391 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.391 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:13:33.391 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.612596+0000" 2024-09-22T03:13:33.391 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.391 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.391 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.391 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.391 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.391 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.391 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:13:33.391 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.392 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.392 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.392 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.392 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.392 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.392 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.392 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:13:33.392 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.647515+0000" 2024-09-22T03:13:33.392 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.392 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.392 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.392 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.392 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.392 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.393 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:13:33.393 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.393 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:13:33.393 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:13:33.393 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.393 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-22T03:13:33.393 INFO:teuthology.orchestra.run.smithi196.stdout: "type": "client_request", 2024-09-22T03:13:33.393 INFO:teuthology.orchestra.run.smithi196.stdout: "id": 346, 2024-09-22T03:13:33.393 INFO:teuthology.orchestra.run.smithi196.stdout: "detail": { 2024-09-22T03:13:33.393 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.393 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:13:33.393 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.641378+0000" 2024-09-22T03:13:33.393 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.393 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.394 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:13:33.394 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.641388+0000" 2024-09-22T03:13:33.394 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.394 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.394 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.394 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.394 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.394 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.394 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:13:33.394 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.641505+0000" 2024-09-22T03:13:33.394 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.394 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.394 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.395 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.395 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.395 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.395 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:13:33.395 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.395 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.395 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.395 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.395 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.395 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.395 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.396 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:13:33.396 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.641658+0000" 2024-09-22T03:13:33.396 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.396 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.396 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.396 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.396 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.396 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.396 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:13:33.396 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.642056+0000" 2024-09-22T03:13:33.396 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.396 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.396 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.396 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.397 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.397 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.397 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:13:33.397 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.397 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.397 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.397 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.397 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.397 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.397 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.397 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:13:33.397 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.669257+0000" 2024-09-22T03:13:33.397 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.397 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.397 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:13:33.398 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.642408+0000" 2024-09-22T03:13:33.398 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.398 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.398 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.398 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.398 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.398 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.398 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:13:33.398 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.398 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.398 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.398 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.398 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.399 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.399 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.399 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:13:33.399 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.642584+0000" 2024-09-22T03:13:33.399 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.399 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.399 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.399 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.399 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.399 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.399 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:13:33.399 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.399 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.400 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.400 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.400 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.400 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.400 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.400 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:13:33.400 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.642763+0000" 2024-09-22T03:13:33.400 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.400 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.400 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.400 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.400 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.400 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.400 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:13:33.400 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.401 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.401 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.401 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.401 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.401 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.401 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.401 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:13:33.401 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.401 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.401 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.401 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.401 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.401 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.401 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.402 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:13:33.402 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.402 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.402 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.402 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.402 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.402 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.402 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.402 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:13:33.402 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.642992+0000" 2024-09-22T03:13:33.402 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.402 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.402 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.403 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.403 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.403 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.403 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:13:33.403 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.403 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.403 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.403 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.403 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.403 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.403 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.403 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:13:33.403 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.644364+0000" 2024-09-22T03:13:33.403 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.404 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.404 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.404 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.404 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.404 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.404 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:13:33.404 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.404 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.404 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.404 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.404 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.404 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.404 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.404 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:13:33.405 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.668766+0000" 2024-09-22T03:13:33.405 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.405 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.405 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.405 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.405 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.405 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.405 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:13:33.405 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.405 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:13:33.405 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:13:33.405 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.405 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-22T03:13:33.405 INFO:teuthology.orchestra.run.smithi196.stdout: "type": "client_request", 2024-09-22T03:13:33.405 INFO:teuthology.orchestra.run.smithi196.stdout: "id": 348, 2024-09-22T03:13:33.406 INFO:teuthology.orchestra.run.smithi196.stdout: "detail": { 2024-09-22T03:13:33.406 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.406 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:13:33.406 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.646648+0000" 2024-09-22T03:13:33.406 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.406 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.406 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:13:33.406 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.646660+0000" 2024-09-22T03:13:33.406 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.406 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.406 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.406 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.406 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.407 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.407 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:13:33.407 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.646850+0000" 2024-09-22T03:13:33.407 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.407 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.407 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.407 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.407 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.407 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.407 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:13:33.407 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.407 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.407 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.408 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.408 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.408 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.408 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.408 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:13:33.408 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.646959+0000" 2024-09-22T03:13:33.408 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.408 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.408 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.408 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.408 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.408 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.408 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:13:33.408 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.648154+0000" 2024-09-22T03:13:33.409 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.409 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.409 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.409 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.409 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.409 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.409 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:13:33.409 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.409 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.410 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.410 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.410 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.410 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.410 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.410 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:13:33.411 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.677150+0000" 2024-09-22T03:13:33.411 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.411 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.411 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:13:33.411 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.648546+0000" 2024-09-22T03:13:33.411 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.411 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.412 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.412 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.412 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.412 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.412 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:13:33.412 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.413 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.413 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.413 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.413 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.413 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.413 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.413 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:13:33.413 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.648865+0000" 2024-09-22T03:13:33.413 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.413 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.413 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.413 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.413 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.414 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.414 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:13:33.414 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.414 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.414 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.414 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.414 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.414 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.414 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.414 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:13:33.414 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.650827+0000" 2024-09-22T03:13:33.414 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.414 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.415 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.415 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.415 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.415 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.415 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:13:33.415 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.415 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.415 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.415 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.416 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.416 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.416 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.416 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:13:33.416 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.416 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.416 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.416 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.416 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.416 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.416 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.416 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:13:33.416 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.416 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.417 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.417 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.417 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.417 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.417 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.417 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:13:33.417 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.651175+0000" 2024-09-22T03:13:33.417 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.417 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.417 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.417 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.417 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.417 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.417 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:13:33.418 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.418 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.418 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.418 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.418 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.418 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.418 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.418 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:13:33.418 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.654402+0000" 2024-09-22T03:13:33.418 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.418 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.418 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.418 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.418 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.419 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.419 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:13:33.419 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.419 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.419 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.419 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.419 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.419 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.419 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.419 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:13:33.419 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.676689+0000" 2024-09-22T03:13:33.419 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.419 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.419 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.420 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.420 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.420 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.420 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:13:33.420 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.420 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:13:33.420 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:13:33.420 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.420 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-22T03:13:33.420 INFO:teuthology.orchestra.run.smithi196.stdout: "type": "client_request", 2024-09-22T03:13:33.420 INFO:teuthology.orchestra.run.smithi196.stdout: "id": 361, 2024-09-22T03:13:33.420 INFO:teuthology.orchestra.run.smithi196.stdout: "detail": { 2024-09-22T03:13:33.420 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.420 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:13:33.421 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.685757+0000" 2024-09-22T03:13:33.421 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.421 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.421 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:13:33.421 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.685787+0000" 2024-09-22T03:13:33.421 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.421 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.421 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.421 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.421 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.421 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.421 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:13:33.421 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.686173+0000" 2024-09-22T03:13:33.421 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.422 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.422 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.422 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.422 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.422 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.422 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:13:33.423 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.423 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.423 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.423 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.423 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.423 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.424 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.424 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:13:33.424 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.686478+0000" 2024-09-22T03:13:33.424 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.424 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.424 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.425 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.425 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.425 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.425 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:13:33.425 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.688483+0000" 2024-09-22T03:13:33.425 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.425 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.426 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.426 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.426 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.426 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.426 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:13:33.426 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.426 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.426 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.426 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.426 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.426 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.426 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.427 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:13:33.427 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.714971+0000" 2024-09-22T03:13:33.427 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.427 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.427 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:13:33.427 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.688823+0000" 2024-09-22T03:13:33.427 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.427 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.427 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.427 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.427 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.427 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.427 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:13:33.427 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.428 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.428 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.428 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.428 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.428 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.428 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.428 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:13:33.428 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.688992+0000" 2024-09-22T03:13:33.428 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.428 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.428 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.428 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.428 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.428 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.429 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:13:33.429 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.429 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.429 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.429 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.429 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.429 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.429 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.429 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:13:33.429 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.689159+0000" 2024-09-22T03:13:33.429 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.429 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.429 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.429 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.429 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.430 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.430 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:13:33.430 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.430 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.430 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.430 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.430 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.430 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.431 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.431 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:13:33.431 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.431 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.431 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.431 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.431 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.431 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.431 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.431 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:13:33.431 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.431 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.431 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.431 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.431 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.432 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.432 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.432 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:13:33.432 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.689429+0000" 2024-09-22T03:13:33.432 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.432 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.432 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.432 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.432 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.432 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.432 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:13:33.432 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.432 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.432 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.433 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.433 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.433 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.433 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.433 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:13:33.433 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.692183+0000" 2024-09-22T03:13:33.433 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.433 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.433 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.433 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.433 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.433 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.433 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:13:33.433 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.433 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.434 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.434 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.434 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.434 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.434 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.434 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:13:33.434 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.714559+0000" 2024-09-22T03:13:33.434 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.434 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.434 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.434 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.434 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.434 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.434 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:13:33.435 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.435 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:13:33.435 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:13:33.435 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.435 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-22T03:13:33.435 INFO:teuthology.orchestra.run.smithi196.stdout: "type": "client_request", 2024-09-22T03:13:33.435 INFO:teuthology.orchestra.run.smithi196.stdout: "id": 381, 2024-09-22T03:13:33.435 INFO:teuthology.orchestra.run.smithi196.stdout: "detail": { 2024-09-22T03:13:33.435 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.435 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:13:33.435 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.756598+0000" 2024-09-22T03:13:33.435 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.435 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.435 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:13:33.435 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.756609+0000" 2024-09-22T03:13:33.436 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.436 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.436 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.436 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.436 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.436 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.436 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:13:33.436 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.756893+0000" 2024-09-22T03:13:33.436 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.436 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.436 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.436 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.436 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.436 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.437 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:13:33.437 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.437 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.437 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.437 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.437 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.437 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.437 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.437 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:13:33.437 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.757267+0000" 2024-09-22T03:13:33.437 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.437 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.437 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.438 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.438 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.438 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.438 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:13:33.438 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.759146+0000" 2024-09-22T03:13:33.438 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.438 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.438 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.438 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.438 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.438 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.439 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:13:33.439 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.439 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.439 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.439 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.439 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.439 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.439 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.439 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:13:33.439 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.790742+0000" 2024-09-22T03:13:33.439 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.440 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.440 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:13:33.440 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.759497+0000" 2024-09-22T03:13:33.440 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.440 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.440 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.440 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.440 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.440 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.440 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:13:33.440 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.440 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.440 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.441 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.441 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.441 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.441 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.441 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:13:33.441 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.759662+0000" 2024-09-22T03:13:33.441 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.441 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.441 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.441 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.441 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.441 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.441 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:13:33.441 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.442 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.442 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.442 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.442 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.442 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.442 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.442 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:13:33.442 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.759841+0000" 2024-09-22T03:13:33.442 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.442 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.442 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.442 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.442 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.443 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.443 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:13:33.443 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.443 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.443 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.443 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.443 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.443 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.443 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.443 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:13:33.443 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.443 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.443 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.443 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.444 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.444 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.444 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.444 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:13:33.444 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.444 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.444 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.444 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.444 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.444 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.444 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.444 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:13:33.444 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.760096+0000" 2024-09-22T03:13:33.444 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.444 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.445 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.445 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.445 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.445 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.445 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:13:33.445 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.445 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.445 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.445 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.445 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.445 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.445 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.445 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:13:33.445 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.761361+0000" 2024-09-22T03:13:33.446 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.446 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.446 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.446 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.446 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.446 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.446 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:13:33.446 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.446 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.446 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.446 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.446 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.446 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.446 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.446 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:13:33.447 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.787925+0000" 2024-09-22T03:13:33.447 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.447 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.447 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.447 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.447 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.447 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.447 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:13:33.447 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.447 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:13:33.447 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:13:33.447 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.447 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-22T03:13:33.447 INFO:teuthology.orchestra.run.smithi196.stdout: "type": "client_request", 2024-09-22T03:13:33.448 INFO:teuthology.orchestra.run.smithi196.stdout: "id": 385, 2024-09-22T03:13:33.448 INFO:teuthology.orchestra.run.smithi196.stdout: "detail": { 2024-09-22T03:13:33.448 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.448 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:13:33.448 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.772247+0000" 2024-09-22T03:13:33.448 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.448 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.448 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:13:33.448 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.772253+0000" 2024-09-22T03:13:33.448 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.448 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.448 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.448 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.448 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.449 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.449 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:13:33.449 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.772444+0000" 2024-09-22T03:13:33.449 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.449 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.449 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.449 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.449 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.449 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.449 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:13:33.449 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.449 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.449 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.449 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.450 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.450 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.450 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.450 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:13:33.450 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.772865+0000" 2024-09-22T03:13:33.450 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.450 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.450 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.450 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.450 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.450 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.450 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:13:33.450 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.773366+0000" 2024-09-22T03:13:33.450 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.451 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.451 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.451 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.451 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.451 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.451 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:13:33.451 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.451 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.451 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.451 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.451 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.451 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.451 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.451 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:13:33.452 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.804930+0000" 2024-09-22T03:13:33.452 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.452 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.452 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:13:33.452 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.773659+0000" 2024-09-22T03:13:33.452 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.452 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.452 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.452 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.452 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.452 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.452 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:13:33.452 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.452 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.453 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.453 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.453 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.453 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.453 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.453 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:13:33.453 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.773845+0000" 2024-09-22T03:13:33.453 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.453 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.453 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.453 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.453 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.453 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.453 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:13:33.453 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.454 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.454 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.454 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.454 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.454 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.454 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.454 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:13:33.454 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.774001+0000" 2024-09-22T03:13:33.454 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.454 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.454 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.454 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.454 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.454 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.455 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:13:33.455 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.455 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.455 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.455 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.455 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.455 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.455 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.455 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:13:33.455 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.455 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.455 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.455 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.455 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.456 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.456 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.456 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:13:33.456 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.456 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.456 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.456 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.456 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.456 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.456 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.456 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:13:33.456 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.774253+0000" 2024-09-22T03:13:33.456 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.456 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.456 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.457 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.457 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.457 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.457 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:13:33.457 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.457 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.457 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.457 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.457 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.457 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.457 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.457 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:13:33.457 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.775592+0000" 2024-09-22T03:13:33.457 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.457 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.458 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.458 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.458 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.458 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.458 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:13:33.458 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.458 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.458 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.458 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.458 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.458 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.458 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.458 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:13:33.458 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.804492+0000" 2024-09-22T03:13:33.459 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.459 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.459 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.459 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.459 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.459 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.459 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:13:33.459 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.459 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:13:33.459 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:13:33.459 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.459 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-22T03:13:33.459 INFO:teuthology.orchestra.run.smithi196.stdout: "type": "client_request", 2024-09-22T03:13:33.459 INFO:teuthology.orchestra.run.smithi196.stdout: "id": 388, 2024-09-22T03:13:33.460 INFO:teuthology.orchestra.run.smithi196.stdout: "detail": { 2024-09-22T03:13:33.460 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.460 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:13:33.460 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.783892+0000" 2024-09-22T03:13:33.460 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.460 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.460 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:13:33.460 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.783900+0000" 2024-09-22T03:13:33.460 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.460 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.460 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.460 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.460 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.460 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.461 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:13:33.461 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.784064+0000" 2024-09-22T03:13:33.461 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.461 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.461 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.461 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.461 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.461 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.461 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:13:33.461 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.461 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.461 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.461 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.461 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.461 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.462 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.462 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:13:33.462 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.784202+0000" 2024-09-22T03:13:33.462 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.462 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.462 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.462 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.462 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.462 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.462 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:13:33.462 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.785551+0000" 2024-09-22T03:13:33.462 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.462 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.462 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.463 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.463 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.463 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.463 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:13:33.463 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.463 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.463 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.463 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.463 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.463 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.463 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.463 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:13:33.463 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.822897+0000" 2024-09-22T03:13:33.463 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.463 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.464 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:13:33.464 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.785866+0000" 2024-09-22T03:13:33.464 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.464 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.464 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.464 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.464 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.464 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.464 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:13:33.464 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.464 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.464 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.464 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.464 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.465 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.465 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.465 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:13:33.465 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.786020+0000" 2024-09-22T03:13:33.465 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.465 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.465 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.465 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.465 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.465 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.465 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:13:33.465 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.465 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.465 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.466 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.466 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.466 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.466 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.466 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:13:33.466 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.786192+0000" 2024-09-22T03:13:33.466 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.466 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.466 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.466 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.466 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.466 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.466 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:13:33.466 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.467 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.467 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.467 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.467 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.467 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.467 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.467 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:13:33.467 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.467 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.467 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.467 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.467 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.467 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.467 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.467 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:13:33.468 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.468 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.468 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.468 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.468 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.468 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.468 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.468 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:13:33.468 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.786426+0000" 2024-09-22T03:13:33.468 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.468 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.468 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.468 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.468 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.469 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.469 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:13:33.469 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.469 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.469 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.469 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.469 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.469 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.469 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.469 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:13:33.469 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.793708+0000" 2024-09-22T03:13:33.469 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.469 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.469 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.469 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.470 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.470 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.470 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:13:33.470 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.470 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.470 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.470 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.470 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.470 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.470 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.470 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:13:33.470 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.822387+0000" 2024-09-22T03:13:33.470 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.470 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.471 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.471 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.471 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.471 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.471 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:13:33.471 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.471 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:13:33.471 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:13:33.471 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.471 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-22T03:13:33.471 INFO:teuthology.orchestra.run.smithi196.stdout: "type": "client_request", 2024-09-22T03:13:33.471 INFO:teuthology.orchestra.run.smithi196.stdout: "id": 390, 2024-09-22T03:13:33.471 INFO:teuthology.orchestra.run.smithi196.stdout: "detail": { 2024-09-22T03:13:33.471 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.471 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:13:33.472 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.786475+0000" 2024-09-22T03:13:33.472 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.472 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.472 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:13:33.472 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.786480+0000" 2024-09-22T03:13:33.472 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.472 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.472 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.472 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.472 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.472 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.472 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:13:33.472 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.786546+0000" 2024-09-22T03:13:33.472 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.472 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.473 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.473 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.473 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.473 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.473 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:13:33.473 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.473 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.473 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.473 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.473 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.473 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.473 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.473 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:13:33.473 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.786717+0000" 2024-09-22T03:13:33.473 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.474 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.474 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.474 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.474 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.474 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.474 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:13:33.474 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.787035+0000" 2024-09-22T03:13:33.474 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.474 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.474 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.474 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.474 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.474 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.475 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:13:33.475 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.475 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.475 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.475 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.475 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.475 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.475 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.475 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:13:33.475 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.836090+0000" 2024-09-22T03:13:33.475 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.475 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.475 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:13:33.475 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.787557+0000" 2024-09-22T03:13:33.475 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.476 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.476 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.476 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.476 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.476 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.476 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:13:33.476 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.476 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.476 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.476 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.476 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.476 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.476 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.476 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:13:33.476 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.789893+0000" 2024-09-22T03:13:33.477 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.477 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.477 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.477 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.477 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.477 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.477 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:13:33.477 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.477 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.477 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.477 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.477 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.477 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.477 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.478 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:13:33.478 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.790086+0000" 2024-09-22T03:13:33.478 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.478 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.478 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.478 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.478 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.478 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.478 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:13:33.478 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.478 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.478 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.478 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.478 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.478 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.479 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.479 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:13:33.479 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.479 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.479 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.479 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.479 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.479 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.479 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.479 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:13:33.479 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.479 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.479 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.479 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.480 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.480 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.480 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.480 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:13:33.480 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.790396+0000" 2024-09-22T03:13:33.480 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.480 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.480 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.480 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.480 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.480 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.480 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:13:33.480 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.480 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.480 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.481 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.481 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.481 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.481 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.481 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:13:33.481 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.797264+0000" 2024-09-22T03:13:33.481 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.481 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.481 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.481 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.481 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.481 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.481 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:13:33.481 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.481 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.482 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.482 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.482 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.482 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.482 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.482 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:13:33.482 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.835657+0000" 2024-09-22T03:13:33.482 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.482 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.482 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.482 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.482 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.482 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.482 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:13:33.482 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.483 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:13:33.483 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:13:33.483 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.483 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-22T03:13:33.483 INFO:teuthology.orchestra.run.smithi196.stdout: "type": "client_request", 2024-09-22T03:13:33.483 INFO:teuthology.orchestra.run.smithi196.stdout: "id": 394, 2024-09-22T03:13:33.483 INFO:teuthology.orchestra.run.smithi196.stdout: "detail": { 2024-09-22T03:13:33.483 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.483 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:13:33.483 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.797111+0000" 2024-09-22T03:13:33.483 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.483 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.483 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:13:33.483 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.797121+0000" 2024-09-22T03:13:33.484 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.484 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.484 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.484 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.484 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.484 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.484 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:13:33.484 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.799011+0000" 2024-09-22T03:13:33.484 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.484 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.484 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.484 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.484 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.484 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.484 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:13:33.485 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.485 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.485 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.485 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.485 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.485 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.485 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.485 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:13:33.485 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.799433+0000" 2024-09-22T03:13:33.485 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.485 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.485 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.485 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.485 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.486 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.486 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:13:33.486 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.799693+0000" 2024-09-22T03:13:33.486 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.486 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.486 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.486 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.486 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.486 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.486 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:13:33.486 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.486 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.486 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.486 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.487 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.487 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.487 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.487 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:13:33.487 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.863792+0000" 2024-09-22T03:13:33.487 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.487 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.487 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:13:33.487 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.800536+0000" 2024-09-22T03:13:33.487 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.487 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.487 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.487 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.487 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.488 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.488 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:13:33.488 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.488 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.488 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.488 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.488 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.488 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.488 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.488 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:13:33.488 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.803303+0000" 2024-09-22T03:13:33.488 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.488 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.488 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.488 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.489 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.489 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.489 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:13:33.489 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.489 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.489 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.489 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.489 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.489 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.489 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.489 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:13:33.489 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.803563+0000" 2024-09-22T03:13:33.489 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.490 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.490 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.490 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.490 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.490 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.490 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:13:33.490 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.490 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.490 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.490 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.490 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.490 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.490 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.490 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:13:33.490 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.491 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.491 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.491 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.491 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.491 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.491 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.491 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:13:33.491 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.491 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.491 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.491 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.491 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.491 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.492 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.492 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:13:33.492 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.803795+0000" 2024-09-22T03:13:33.492 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.492 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.492 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.492 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.492 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.492 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.492 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:13:33.492 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.492 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.492 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.492 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.493 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.493 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.493 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.493 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:13:33.493 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.805469+0000" 2024-09-22T03:13:33.493 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.493 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.493 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.493 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.493 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.493 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.493 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:13:33.493 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.493 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.494 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.494 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.494 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.494 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.494 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.494 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:13:33.494 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.860792+0000" 2024-09-22T03:13:33.494 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.494 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.494 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.494 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.494 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.494 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.494 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:13:33.494 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.495 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:13:33.495 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:13:33.495 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.495 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-22T03:13:33.495 INFO:teuthology.orchestra.run.smithi196.stdout: "type": "client_request", 2024-09-22T03:13:33.495 INFO:teuthology.orchestra.run.smithi196.stdout: "id": 407, 2024-09-22T03:13:33.495 INFO:teuthology.orchestra.run.smithi196.stdout: "detail": { 2024-09-22T03:13:33.495 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.495 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:13:33.495 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.854894+0000" 2024-09-22T03:13:33.495 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.495 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.495 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:13:33.495 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.854904+0000" 2024-09-22T03:13:33.496 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.496 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.496 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.496 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.496 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.496 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.496 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:13:33.496 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.855198+0000" 2024-09-22T03:13:33.496 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.496 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.496 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.496 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.496 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.496 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.496 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:13:33.497 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.497 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.497 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.497 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.497 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.497 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.497 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.497 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:13:33.497 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.856090+0000" 2024-09-22T03:13:33.497 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.497 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.497 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.497 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.497 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.498 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.498 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:13:33.498 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.856462+0000" 2024-09-22T03:13:33.498 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.498 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.498 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.498 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.498 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.498 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.498 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:13:33.498 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.498 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.498 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.498 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.499 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.499 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.499 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.499 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:13:33.499 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.889696+0000" 2024-09-22T03:13:33.499 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.499 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.499 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:13:33.499 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.856837+0000" 2024-09-22T03:13:33.499 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.499 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.499 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.499 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.499 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.500 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.500 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:13:33.500 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.500 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.500 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.500 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.500 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.500 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.500 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.500 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:13:33.500 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.857013+0000" 2024-09-22T03:13:33.500 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.500 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.500 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.501 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.501 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.501 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.501 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:13:33.501 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.501 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.501 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.501 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.501 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.501 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.501 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.501 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:13:33.501 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.857174+0000" 2024-09-22T03:13:33.501 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.502 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.502 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.502 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.502 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.502 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.502 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:13:33.502 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.502 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.502 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.502 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.502 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.502 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.502 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.502 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:13:33.503 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.503 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.503 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.503 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.503 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.503 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.503 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.503 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:13:33.503 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.503 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.503 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.503 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.503 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.503 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.504 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.504 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:13:33.504 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.857405+0000" 2024-09-22T03:13:33.504 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.504 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.504 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.504 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.504 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.504 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.504 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:13:33.504 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.504 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.504 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.504 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.504 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.505 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.505 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.505 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:13:33.505 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.859252+0000" 2024-09-22T03:13:33.505 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.505 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.505 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.505 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.505 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.505 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.505 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:13:33.505 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.505 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.505 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.505 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.506 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.506 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.506 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.506 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:13:33.506 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.889248+0000" 2024-09-22T03:13:33.506 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.506 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.506 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.506 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.506 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.506 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.506 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:13:33.506 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.506 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:13:33.507 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:13:33.507 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.507 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-22T03:13:33.507 INFO:teuthology.orchestra.run.smithi196.stdout: "type": "client_request", 2024-09-22T03:13:33.507 INFO:teuthology.orchestra.run.smithi196.stdout: "id": 409, 2024-09-22T03:13:33.507 INFO:teuthology.orchestra.run.smithi196.stdout: "detail": { 2024-09-22T03:13:33.507 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.507 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:13:33.507 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.857372+0000" 2024-09-22T03:13:33.507 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.507 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.507 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:13:33.507 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.857377+0000" 2024-09-22T03:13:33.507 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.508 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.508 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.508 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.508 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.508 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.508 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:13:33.508 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.857513+0000" 2024-09-22T03:13:33.508 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.508 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.508 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.508 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.508 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.508 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.508 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:13:33.508 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.509 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.509 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.509 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.509 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.509 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.509 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.509 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:13:33.509 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.857656+0000" 2024-09-22T03:13:33.509 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.509 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.509 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.509 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.509 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.509 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.510 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:13:33.510 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.857920+0000" 2024-09-22T03:13:33.510 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.510 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.510 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.510 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.510 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.510 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.510 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:13:33.510 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.510 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.510 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.510 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.510 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.510 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.511 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.511 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:13:33.511 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.899300+0000" 2024-09-22T03:13:33.511 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.511 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.511 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:13:33.511 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.858305+0000" 2024-09-22T03:13:33.511 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.511 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.511 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.511 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.511 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.511 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.511 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:13:33.512 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.512 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.512 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.512 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.512 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.512 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.512 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.512 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:13:33.512 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.858616+0000" 2024-09-22T03:13:33.512 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.512 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.512 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.512 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.512 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.512 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.513 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:13:33.513 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.513 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.513 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.513 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.513 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.513 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.513 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.513 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:13:33.513 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.858794+0000" 2024-09-22T03:13:33.513 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.513 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.513 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.513 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.514 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.514 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.514 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:13:33.514 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.514 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.514 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.514 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.514 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.514 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.514 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.514 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:13:33.514 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.514 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.514 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.515 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.515 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.515 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.515 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.515 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:13:33.515 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.515 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.515 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.515 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.515 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.515 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.515 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.515 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:13:33.515 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.859731+0000" 2024-09-22T03:13:33.516 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.516 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.516 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.516 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.516 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.516 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.516 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:13:33.516 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.516 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.516 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.516 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.516 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.516 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.516 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.516 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:13:33.517 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.867011+0000" 2024-09-22T03:13:33.517 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.517 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.517 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.517 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.517 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.517 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.517 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:13:33.517 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.517 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.517 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.517 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.517 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.517 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.518 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.518 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:13:33.518 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.899052+0000" 2024-09-22T03:13:33.518 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.518 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.518 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.518 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.518 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.518 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.518 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:13:33.518 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.518 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:13:33.518 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:13:33.518 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.519 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-22T03:13:33.519 INFO:teuthology.orchestra.run.smithi196.stdout: "type": "client_request", 2024-09-22T03:13:33.519 INFO:teuthology.orchestra.run.smithi196.stdout: "id": 412, 2024-09-22T03:13:33.519 INFO:teuthology.orchestra.run.smithi196.stdout: "detail": { 2024-09-22T03:13:33.519 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.519 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:13:33.519 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.873195+0000" 2024-09-22T03:13:33.519 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.519 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.519 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:13:33.519 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.873206+0000" 2024-09-22T03:13:33.519 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.519 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.519 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.520 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.520 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.520 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.520 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:13:33.520 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.874041+0000" 2024-09-22T03:13:33.520 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.520 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.520 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.520 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.520 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.520 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.520 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:13:33.520 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.520 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.520 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.521 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.521 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.521 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.521 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.521 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:13:33.521 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.874224+0000" 2024-09-22T03:13:33.521 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.521 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.521 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.521 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.521 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.521 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.521 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:13:33.521 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.874514+0000" 2024-09-22T03:13:33.522 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.522 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.522 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.522 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.522 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.522 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.522 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:13:33.522 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.522 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.522 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.522 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.522 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.522 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.522 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.523 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:13:33.523 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.901080+0000" 2024-09-22T03:13:33.523 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.523 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.523 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:13:33.523 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.874917+0000" 2024-09-22T03:13:33.523 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.523 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.523 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.523 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.523 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.523 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.523 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:13:33.523 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.523 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.524 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.524 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.524 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.524 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.524 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.524 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:13:33.524 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.875097+0000" 2024-09-22T03:13:33.524 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.524 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.524 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.524 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.524 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.524 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.524 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:13:33.525 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.525 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.525 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.525 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.525 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.525 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.525 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.525 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:13:33.525 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.875258+0000" 2024-09-22T03:13:33.525 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.525 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.525 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.525 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.525 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.525 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.526 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:13:33.526 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.526 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.526 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.526 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.526 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.526 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.526 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.526 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:13:33.526 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.526 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.526 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.526 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.527 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.527 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.527 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.527 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:13:33.527 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.527 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.527 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.527 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.527 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.527 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.527 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.527 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:13:33.527 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.875500+0000" 2024-09-22T03:13:33.527 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.527 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.528 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.528 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.528 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.528 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.528 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:13:33.528 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.528 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.528 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.528 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.528 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.528 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.528 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.528 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:13:33.528 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.877007+0000" 2024-09-22T03:13:33.528 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.529 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.529 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.529 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.529 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.529 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.529 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:13:33.529 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.529 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.529 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.529 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.529 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.529 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.529 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:13:33.529 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:13:33.530 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:13:32.900683+0000" 2024-09-22T03:13:33.530 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.530 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.530 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:13:33.530 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.530 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.530 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:13:33.530 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:13:33.530 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:13:33.530 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:13:33.530 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:13:33.530 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:13:33.530 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-22T03:13:33.531 INFO:teuthology.orchestra.run.smithi196.stdout: "type": "replicated_request", 2024-09-22T03:13:33.531 INFO:teuthology.orchestra.run.smithi196.stdout: "id": 438, 2024-09-22T03:13:33.531 INFO:teuthology.orchestra.run.smithi196.stdout: "detail": { 2024-09-22T03:13:33.531 INFO:teuthology.orchestra.run.smithi196.stdout: "RepRequest": { 2024-09-22T03:13:33.531 INFO:teuthology.orchestra.run.smithi196.stdout: "reqid": "client.4468.0:8354", 2024-09-22T03:13:33.531 INFO:teuthology.orchestra.run.smithi196.stdout: "pgid": "5.d", 2024-09-22T03:13:33.531 INFO:teuthology.orchestra.run.smithi196.stdout: "map_epoch": 60, 2024-09-22T03:13:33.531 INFO:teuthology.orchestra.run.smithi196.stdout: "min_epoch": 59, 2024-09-22T03:13:33.531 INFO:teuthology.orchestra.run.smithi196.stdout: "oid": "5:b63c9745:::benchmark_data_smithi153_33956_object412:head", 2024-09-22T03:13:33.531 INFO:teuthology.orchestra.run.smithi196.stdout: "from": "2" 2024-09-22T03:13:33.531 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:13:33.531 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:13:33.531 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:13:33.531 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-22T03:13:33.532 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-09-22T03:13:34.368 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:Removed 993 objects 2024-09-22T03:13:34.369 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:Clean up completed and total clean up time :2.36603 2024-09-22T03:13:34.537 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-22T03:13:34.641 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:13:34.639+0000 7faebf4d7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:13:34.644 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:13:34.643+0000 7faebf4d7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:13:34.648 INFO:teuthology.orchestra.run.smithi153.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-22T03:13:34.690 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-22T03:13:34.690 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-22T03:13:35.639 INFO:tasks.ceph.ceph_manager.ceph:removing pool_name unique_pool_2 2024-09-22T03:13:35.639 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm unique_pool_2 unique_pool_2 --yes-i-really-really-mean-it 2024-09-22T03:13:35.750 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:13:35.741+0000 7f4a9fd97640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:13:35.753 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:13:35.751+0000 7f4a9fd97640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:13:36.691 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-22T03:13:36.799 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:13:36.796+0000 7f5119e6c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:13:36.802 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:13:36.800+0000 7f5119e6c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:13:36.806 INFO:teuthology.orchestra.run.smithi153.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-22T03:13:36.851 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-22T03:13:36.851 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-22T03:13:36.913 INFO:teuthology.orchestra.run.smithi153.stderr:pool 'unique_pool_2' removed 2024-09-22T03:13:36.927 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-09-22T03:13:36.928 INFO:tasks.radosbench:Beginning radosbench... 2024-09-22T03:13:36.928 INFO:tasks.ceph.ceph_manager.ceph:creating pool_name unique_pool_3 2024-09-22T03:13:36.928 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create unique_pool_3 16 2024-09-22T03:13:37.058 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:13:37.050+0000 7f687aad7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:13:37.061 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:13:37.060+0000 7f687aad7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:13:37.914 INFO:teuthology.orchestra.run.smithi153.stderr:pool 'unique_pool_3' created 2024-09-22T03:13:37.931 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool application enable unique_pool_3 rados --yes-i-really-mean-it || true 2024-09-22T03:13:38.045 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:13:38.035+0000 7f856e997640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:13:38.048 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:13:38.047+0000 7f856e997640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:13:38.533 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [0] live_osds: [2, 1, 3] 2024-09-22T03:13:38.533 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-22T03:13:38.533 INFO:tasks.thrashosds.thrasher:Killing osd 2, live_osds are [2, 1, 3] 2024-09-22T03:13:38.534 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-09-22T03:13:38.534 INFO:teuthology.orchestra.run:waiting for 300 2024-09-22T03:13:38.852 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-22T03:13:38.924 INFO:teuthology.orchestra.run.smithi153.stderr:enabled application 'rados' on pool 'unique_pool_3' 2024-09-22T03:13:38.975 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:13:38.972+0000 7f537af4e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:13:38.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:13:38.978+0000 7f537af4e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:13:38.986 INFO:teuthology.orchestra.run.smithi153.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-22T03:13:39.438 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~0s 2024-09-22T03:13:39.943 DEBUG:teuthology.orchestra.run.smithi153:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 8192 --object-size 65536 -p unique_pool_3 bench 90 write' 2024-09-22T03:13:39.947 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-09-22T03:13:39.947 INFO:teuthology.orchestra.run:waiting for 3000 2024-09-22T03:13:40.007 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:hints = 1 2024-09-22T03:13:40.007 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:Maintaining 16 concurrent writes of 8192 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-09-22T03:13:40.008 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:Object prefix: benchmark_data_smithi153_35916 2024-09-22T03:13:40.008 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-22T03:13:40.008 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 0 0 0 0 0 0 - 0 2024-09-22T03:13:41.007 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 1 16 24 8 0.0624928 0.0625 0.0326998 0.0258077 2024-09-22T03:13:42.007 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 2 16 24 8 0.0312461 0 - 0.0258077 2024-09-22T03:13:43.007 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 3 16 24 8 0.0208308 0 - 0.0258077 2024-09-22T03:13:44.008 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 4 16 24 8 0.0156231 0 - 0.0258077 2024-09-22T03:13:44.634 INFO:tasks.ceph.osd.2:Stopped 2024-09-22T03:13:44.842 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~5s 2024-09-22T03:13:45.008 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 5 16 24 8 0.0124985 0 - 0.0258077 2024-09-22T03:13:46.008 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 6 16 24 8 0.0104154 0 - 0.0258077 2024-09-22T03:13:47.008 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 7 16 24 8 0.0089275 0 - 0.0258077 2024-09-22T03:13:48.008 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 8 16 24 8 0.00781155 0 - 0.0258077 2024-09-22T03:13:49.008 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 9 16 24 8 0.00694357 0 - 0.0258077 2024-09-22T03:13:49.636 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [0, 2] live_osds: [1, 3] 2024-09-22T03:13:49.636 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-22T03:13:49.636 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-09-22T03:13:49.636 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-09-22T03:13:49.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:13:49.747+0000 7fd82766c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:13:49.757 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:13:49.754+0000 7fd82766c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:13:49.760 INFO:teuthology.orchestra.run.smithi153.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-22T03:13:49.764 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-22T03:13:49.764 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-22T03:13:49.764 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-09-22T03:13:49.870 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:13:49.863+0000 7f0b2f2ce640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:13:49.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:13:49.870+0000 7f0b2f2ce640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:13:49.885 INFO:teuthology.orchestra.run.smithi153.stderr:no valid command found; 10 closest matches: 2024-09-22T03:13:49.885 INFO:teuthology.orchestra.run.smithi153.stderr:config get 2024-09-22T03:13:49.885 INFO:teuthology.orchestra.run.smithi153.stderr:config help 2024-09-22T03:13:49.885 INFO:teuthology.orchestra.run.smithi153.stderr:config set ... 2024-09-22T03:13:49.885 INFO:teuthology.orchestra.run.smithi153.stderr:config show 2024-09-22T03:13:49.885 INFO:teuthology.orchestra.run.smithi153.stderr:deep_scrub 2024-09-22T03:13:49.885 INFO:teuthology.orchestra.run.smithi153.stderr:dump_historic_ops 2024-09-22T03:13:49.885 INFO:teuthology.orchestra.run.smithi153.stderr:dump_historic_slow_ops 2024-09-22T03:13:49.886 INFO:teuthology.orchestra.run.smithi153.stderr:dump_metrics [] 2024-09-22T03:13:49.886 INFO:teuthology.orchestra.run.smithi153.stderr:dump_ops_in_flight 2024-09-22T03:13:49.886 INFO:teuthology.orchestra.run.smithi153.stderr:dump_pgstate_history 2024-09-22T03:13:49.886 INFO:teuthology.orchestra.run.smithi153.stderr:Error EINVAL: invalid command 2024-09-22T03:13:49.888 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-22T03:13:49.888 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-22T03:13:49.888 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-09-22T03:13:49.993 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:13:49.986+0000 7f8f8cad7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:13:49.996 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:13:49.994+0000 7f8f8cad7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:13:50.008 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 10 16 24 8 0.00624925 0 - 0.0258077 2024-09-22T03:13:50.245 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 has been restored 2024-09-22T03:13:51.008 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 11 16 24 8 0.00568113 0 - 0.0258077 2024-09-22T03:13:52.009 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 12 16 24 8 0.0052077 0 - 0.0258077 2024-09-22T03:13:53.009 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 13 16 24 8 0.00480711 0 - 0.0258077 2024-09-22T03:13:54.009 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 14 16 24 8 0.00446374 0 - 0.0258077 2024-09-22T03:13:55.009 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 15 16 24 8 0.00416616 0 - 0.0258077 2024-09-22T03:13:56.009 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 16 16 24 8 0.00390577 0 - 0.0258077 2024-09-22T03:13:57.009 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 17 16 24 8 0.00367603 0 - 0.0258077 2024-09-22T03:13:58.009 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 18 16 24 8 0.0034718 0 - 0.0258077 2024-09-22T03:13:59.010 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 19 16 24 8 0.00328907 0 - 0.0258077 2024-09-22T03:14:00.010 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:2024-09-22T03:14:00.009166+0000 min lat: 0.0162411 max lat: 0.0326998 avg lat: 0.0258077 2024-09-22T03:14:00.010 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-22T03:14:00.010 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 20 16 24 8 0.00312462 0 - 0.0258077 2024-09-22T03:14:01.010 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 21 16 24 8 0.00297582 0 - 0.0258077 2024-09-22T03:14:02.010 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 22 16 24 8 0.00284055 0 - 0.0258077 2024-09-22T03:14:03.010 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 23 16 24 8 0.00271705 0 - 0.0258077 2024-09-22T03:14:04.010 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 24 16 24 8 0.00260384 0 - 0.0258077 2024-09-22T03:14:05.010 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 25 16 24 8 0.00249968 0 - 0.0258077 2024-09-22T03:14:05.893 INFO:teuthology.orchestra.run.smithi153.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-09-22T03:14:05.893 INFO:teuthology.orchestra.run.smithi153.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-09-22T03:14:05.897 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-22T03:14:05.898 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-22T03:14:05.898 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-09-22T03:14:05.956 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-22T03:14:05.957 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-22T03:14:06.010 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 26 16 24 8 0.00240354 0 - 0.0258077 2024-09-22T03:14:06.019 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:14:06.013+0000 7efde8d97640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:14:06.022 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:14:06.021+0000 7efde8d97640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:14:06.035 INFO:teuthology.orchestra.run.smithi153.stderr:no valid command found; 10 closest matches: 2024-09-22T03:14:06.035 INFO:teuthology.orchestra.run.smithi153.stderr:config get 2024-09-22T03:14:06.035 INFO:teuthology.orchestra.run.smithi153.stderr:config help 2024-09-22T03:14:06.035 INFO:teuthology.orchestra.run.smithi153.stderr:config set ... 2024-09-22T03:14:06.035 INFO:teuthology.orchestra.run.smithi153.stderr:config show 2024-09-22T03:14:06.035 INFO:teuthology.orchestra.run.smithi153.stderr:deep_scrub 2024-09-22T03:14:06.035 INFO:teuthology.orchestra.run.smithi153.stderr:dump_historic_ops 2024-09-22T03:14:06.036 INFO:teuthology.orchestra.run.smithi153.stderr:dump_historic_slow_ops 2024-09-22T03:14:06.036 INFO:teuthology.orchestra.run.smithi153.stderr:dump_metrics [] 2024-09-22T03:14:06.036 INFO:teuthology.orchestra.run.smithi153.stderr:dump_ops_in_flight 2024-09-22T03:14:06.036 INFO:teuthology.orchestra.run.smithi153.stderr:dump_pgstate_history 2024-09-22T03:14:06.036 INFO:teuthology.orchestra.run.smithi153.stderr:Error EINVAL: invalid command 2024-09-22T03:14:06.038 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-22T03:14:06.039 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-22T03:14:07.010 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 27 16 33 17 0.00491837 0.00270433 26.9961 14.2954 2024-09-22T03:14:07.958 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-22T03:14:08.011 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 28 16 41 25 0.00697459 0.0625 0.00703239 9.72339 2024-09-22T03:14:08.070 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:14:08.067+0000 7f22a8157640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:14:08.073 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:14:08.071+0000 7f22a8157640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:14:08.076 INFO:teuthology.orchestra.run.smithi153.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-22T03:14:08.090 INFO:teuthology.orchestra.run.smithi153.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-09-22T03:14:08.109 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-22T03:14:08.109 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-22T03:14:09.011 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 29 16 41 25 0.00673408 0 - 9.72339 2024-09-22T03:14:10.011 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 30 16 41 25 0.00650962 0 - 9.72339 2024-09-22T03:14:10.109 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-22T03:14:10.229 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:14:10.225+0000 7faeb1597640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:14:10.232 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:14:10.230+0000 7faeb1597640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:14:10.236 INFO:teuthology.orchestra.run.smithi153.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-22T03:14:10.249 INFO:teuthology.orchestra.run.smithi153.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-09-22T03:14:10.268 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-22T03:14:10.269 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-22T03:14:11.011 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 31 16 41 25 0.00629963 0 - 9.72339 2024-09-22T03:14:11.040 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [0, 2] live_osds: [1, 3] 2024-09-22T03:14:11.040 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-22T03:14:11.040 INFO:tasks.thrashosds.thrasher:Reviving osd 2 2024-09-22T03:14:11.040 INFO:tasks.ceph.osd.2:Restarting daemon 2024-09-22T03:14:11.040 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2>> /var/log/ceph/ceph-osd.2.log 2024-09-22T03:14:11.043 INFO:tasks.ceph.osd.2:Started 2024-09-22T03:14:11.043 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-22T03:14:11.154 INFO:teuthology.orchestra.run.smithi196.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-09-22T03:14:11.157 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-22T03:14:11.158 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2024-09-22T03:14:12.011 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 32 16 41 25 0.00610277 0 - 9.72339 2024-09-22T03:14:12.269 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-22T03:14:12.385 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:14:12.382+0000 7fd43bd0e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:14:12.389 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:14:12.387+0000 7fd43bd0e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:14:12.392 INFO:teuthology.orchestra.run.smithi153.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-22T03:14:12.406 INFO:teuthology.orchestra.run.smithi153.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-09-22T03:14:12.423 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-22T03:14:12.423 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-22T03:14:13.011 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 33 16 41 25 0.00591783 0 - 9.72339 2024-09-22T03:14:14.011 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 34 16 41 25 0.00574377 0 - 9.72339 2024-09-22T03:14:14.424 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-22T03:14:14.540 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:14:14.537+0000 7f6d5686c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:14:14.543 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:14:14.542+0000 7f6d5686c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:14:14.547 INFO:teuthology.orchestra.run.smithi153.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-22T03:14:14.561 INFO:teuthology.orchestra.run.smithi153.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-09-22T03:14:14.584 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-22T03:14:14.584 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-22T03:14:15.011 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 35 16 41 25 0.00557967 0 - 9.72339 2024-09-22T03:14:16.012 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 36 16 41 25 0.00542467 0 - 9.72339 2024-09-22T03:14:16.159 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-22T03:14:16.256 INFO:teuthology.orchestra.run.smithi196.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-09-22T03:14:16.259 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-22T03:14:16.259 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2024-09-22T03:14:16.585 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-22T03:14:16.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:14:16.697+0000 7ff5ae797640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:14:16.704 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:14:16.702+0000 7ff5ae797640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:14:16.708 INFO:teuthology.orchestra.run.smithi153.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-22T03:14:16.721 INFO:teuthology.orchestra.run.smithi153.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-09-22T03:14:16.739 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-22T03:14:16.739 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-22T03:14:17.012 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 37 16 41 25 0.00527806 0 - 9.72339 2024-09-22T03:14:18.012 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 38 16 41 25 0.00513916 0 - 9.72339 2024-09-22T03:14:18.740 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-22T03:14:18.857 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:14:18.854+0000 7fdef86d7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:14:18.860 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:14:18.858+0000 7fdef86d7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:14:18.864 INFO:teuthology.orchestra.run.smithi153.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-22T03:14:18.877 INFO:teuthology.orchestra.run.smithi153.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-09-22T03:14:18.901 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-22T03:14:18.902 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-22T03:14:19.012 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 39 16 41 25 0.00500739 0 - 9.72339 2024-09-22T03:14:20.012 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:2024-09-22T03:14:20.011766+0000 min lat: 0.00692909 max lat: 26.9961 avg lat: 9.72339 2024-09-22T03:14:20.012 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-22T03:14:20.012 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 40 16 41 25 0.0048822 0 - 9.72339 2024-09-22T03:14:20.902 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-22T03:14:21.012 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 41 16 41 25 0.00476312 0 - 9.72339 2024-09-22T03:14:21.036 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:14:21.033+0000 7fd2eff97640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:14:21.039 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:14:21.037+0000 7fd2eff97640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:14:21.044 INFO:teuthology.orchestra.run.smithi153.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-22T03:14:21.101 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-22T03:14:21.101 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-22T03:14:21.260 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-22T03:14:21.366 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-09-22T03:14:21.366 INFO:teuthology.orchestra.run.smithi196.stdout: "ops_in_flight": [ 2024-09-22T03:14:21.366 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-22T03:14:21.366 INFO:teuthology.orchestra.run.smithi196.stdout: "type": "client_request", 2024-09-22T03:14:21.366 INFO:teuthology.orchestra.run.smithi196.stdout: "id": 27, 2024-09-22T03:14:21.366 INFO:teuthology.orchestra.run.smithi196.stdout: "detail": { 2024-09-22T03:14:21.366 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:14:21.366 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:14:21.367 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:14:21.022467+0000" 2024-09-22T03:14:21.367 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.367 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:14:21.367 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:14:21.367 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:14:21.022498+0000" 2024-09-22T03:14:21.367 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.367 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:14:21.367 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:21.367 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.367 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.367 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:14:21.367 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:14:21.367 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:14:21.027669+0000" 2024-09-22T03:14:21.367 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.367 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:14:21.368 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:21.368 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.368 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.368 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:14:21.368 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:14:21.368 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.368 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.368 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:14:21.368 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:21.368 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.368 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.368 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:14:21.368 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:14:21.368 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:14:21.028257+0000" 2024-09-22T03:14:21.369 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.369 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:14:21.369 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:21.369 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.369 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.369 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:14:21.369 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:14:21.369 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:14:21.028514+0000" 2024-09-22T03:14:21.369 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.369 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:14:21.369 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:21.369 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.369 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.369 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:14:21.369 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:14:21.370 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.370 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.370 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:14:21.370 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:21.370 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.370 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.370 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:14:21.370 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:14:21.370 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.370 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.370 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:14:21.370 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:14:21.370 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:14:21.028907+0000" 2024-09-22T03:14:21.370 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.371 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:14:21.371 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:21.371 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.371 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.371 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:14:21.371 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:14:21.371 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.371 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.371 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:14:21.371 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:21.371 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.371 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.371 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:14:21.371 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:14:21.371 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:14:21.029092+0000" 2024-09-22T03:14:21.371 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.372 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:14:21.372 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:21.372 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.372 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.372 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:14:21.372 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:14:21.372 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:14:21.029309+0000", 2024-09-22T03:14:21.372 INFO:teuthology.orchestra.run.smithi196.stdout: "blocker": { 2024-09-22T03:14:21.372 INFO:teuthology.orchestra.run.smithi196.stdout: "op_type": "PGActivationBlocker", 2024-09-22T03:14:21.372 INFO:teuthology.orchestra.run.smithi196.stdout: "detail": { 2024-09-22T03:14:21.372 INFO:teuthology.orchestra.run.smithi196.stdout: "pgid": "6.f" 2024-09-22T03:14:21.372 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:14:21.372 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:14:21.372 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.372 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:14:21.373 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:21.373 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.373 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.373 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:14:21.373 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:14:21.373 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.373 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.373 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:14:21.373 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:21.373 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.373 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.373 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:14:21.373 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:14:21.373 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.373 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.373 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:14:21.374 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:21.374 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.374 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.374 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:14:21.374 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:14:21.374 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.374 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.374 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:14:21.374 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:21.374 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.374 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.374 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:14:21.374 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:14:21.375 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.375 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.375 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:14:21.375 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:21.375 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.375 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.375 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:14:21.375 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:14:21.375 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.375 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.375 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:14:21.375 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:21.375 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.375 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.375 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:14:21.375 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:14:21.376 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.376 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.376 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:14:21.376 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:21.376 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.376 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.376 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:14:21.376 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:14:21.376 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.376 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.376 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:14:21.376 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:21.376 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.376 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.376 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:14:21.376 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:14:21.376 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.377 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.377 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:14:21.377 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:21.377 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.377 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.377 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:14:21.377 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:14:21.377 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.377 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.377 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:14:21.377 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:21.377 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.377 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.377 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:14:21.377 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:14:21.378 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.378 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:14:21.378 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:14:21.378 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.378 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-22T03:14:21.378 INFO:teuthology.orchestra.run.smithi196.stdout: "type": "client_request", 2024-09-22T03:14:21.378 INFO:teuthology.orchestra.run.smithi196.stdout: "id": 30, 2024-09-22T03:14:21.378 INFO:teuthology.orchestra.run.smithi196.stdout: "detail": { 2024-09-22T03:14:21.378 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:14:21.378 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:14:21.378 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:14:21.030180+0000" 2024-09-22T03:14:21.378 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.378 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:14:21.378 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:14:21.378 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:14:21.030185+0000" 2024-09-22T03:14:21.379 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.379 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:14:21.379 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:21.379 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.379 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.379 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:14:21.379 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:14:21.379 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:14:21.030303+0000" 2024-09-22T03:14:21.379 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.379 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:14:21.379 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:21.379 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.379 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.379 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:14:21.379 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:14:21.379 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.380 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.380 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:14:21.380 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:21.380 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.380 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.380 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:14:21.380 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:14:21.380 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:14:21.030587+0000" 2024-09-22T03:14:21.380 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.380 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:14:21.380 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:21.380 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.380 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.380 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:14:21.380 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:14:21.380 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:14:21.030653+0000" 2024-09-22T03:14:21.380 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.381 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:14:21.381 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:21.381 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.381 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.381 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:14:21.381 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:14:21.381 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.381 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.381 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:14:21.381 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:21.381 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.381 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.381 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:14:21.381 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:14:21.381 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.381 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.382 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:14:21.382 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:14:21.382 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:14:21.031031+0000" 2024-09-22T03:14:21.382 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.382 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:14:21.382 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:21.382 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.382 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.382 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:14:21.382 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:14:21.382 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.382 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.382 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:14:21.382 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:21.382 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.383 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.383 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:14:21.383 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:14:21.383 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:14:21.031295+0000", 2024-09-22T03:14:21.383 INFO:teuthology.orchestra.run.smithi196.stdout: "blocker": { 2024-09-22T03:14:21.383 INFO:teuthology.orchestra.run.smithi196.stdout: "op_type": "CommonPGPipeline:::wait_for_active", 2024-09-22T03:14:21.383 INFO:teuthology.orchestra.run.smithi196.stdout: "detail": { 2024-09-22T03:14:21.383 INFO:teuthology.orchestra.run.smithi196.stdout: "waiting": 1, 2024-09-22T03:14:21.383 INFO:teuthology.orchestra.run.smithi196.stdout: "held_by_operation_id": 27 2024-09-22T03:14:21.383 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:14:21.383 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:14:21.383 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.383 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:14:21.383 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:21.383 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.383 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.384 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:14:21.384 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:14:21.384 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.384 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.384 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:14:21.384 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:21.384 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.384 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.384 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:14:21.384 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:14:21.384 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.384 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.384 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:14:21.384 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:21.384 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.384 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.385 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:14:21.385 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:14:21.385 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.385 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.385 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:14:21.385 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:21.385 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.385 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.385 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:14:21.385 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:14:21.385 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.385 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.385 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:14:21.385 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:21.386 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.386 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.386 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:14:21.386 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:14:21.386 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.386 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.386 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:14:21.386 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:21.386 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.386 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.386 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:14:21.386 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:14:21.386 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.386 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.386 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:14:21.387 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:21.387 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.387 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.387 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:14:21.387 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:14:21.387 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.387 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.387 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:14:21.387 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:21.387 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.387 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.387 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:14:21.387 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:14:21.387 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.387 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.388 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:14:21.388 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:21.388 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.388 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.388 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:14:21.388 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:14:21.388 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.388 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.388 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:14:21.388 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:21.388 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.388 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.388 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:14:21.388 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:14:21.388 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.388 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.389 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:14:21.389 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:21.389 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.389 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.389 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:14:21.389 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:14:21.389 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.389 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:14:21.389 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:14:21.389 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.389 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-22T03:14:21.389 INFO:teuthology.orchestra.run.smithi196.stdout: "type": "client_request", 2024-09-22T03:14:21.389 INFO:teuthology.orchestra.run.smithi196.stdout: "id": 32, 2024-09-22T03:14:21.389 INFO:teuthology.orchestra.run.smithi196.stdout: "detail": { 2024-09-22T03:14:21.389 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:14:21.389 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:14:21.390 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:14:21.031608+0000" 2024-09-22T03:14:21.390 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.390 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:14:21.390 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:14:21.390 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:14:21.031612+0000" 2024-09-22T03:14:21.390 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.390 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:14:21.390 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:21.390 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.390 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.390 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:14:21.390 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:14:21.390 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:14:21.031688+0000" 2024-09-22T03:14:21.390 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.390 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:14:21.390 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:21.391 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.391 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.391 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:14:21.391 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:14:21.391 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.391 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.391 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:14:21.391 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:21.391 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.391 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.391 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:14:21.391 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:14:21.391 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:14:21.031740+0000" 2024-09-22T03:14:21.391 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.392 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:14:21.392 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:21.392 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.392 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.392 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:14:21.392 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:14:21.392 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:14:21.031826+0000" 2024-09-22T03:14:21.392 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.392 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:14:21.392 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:21.392 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.392 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.392 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:14:21.393 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:14:21.393 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.393 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.393 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:14:21.393 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:21.393 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.393 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.393 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:14:21.393 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:14:21.393 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.393 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.393 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:14:21.393 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:14:21.393 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:14:21.032276+0000", 2024-09-22T03:14:21.393 INFO:teuthology.orchestra.run.smithi196.stdout: "blocker": { 2024-09-22T03:14:21.394 INFO:teuthology.orchestra.run.smithi196.stdout: "op_type": "ClientRequest::PGPipeline::await_map", 2024-09-22T03:14:21.394 INFO:teuthology.orchestra.run.smithi196.stdout: "detail": { 2024-09-22T03:14:21.394 INFO:teuthology.orchestra.run.smithi196.stdout: "waiting": 1, 2024-09-22T03:14:21.394 INFO:teuthology.orchestra.run.smithi196.stdout: "held_by_operation_id": 30 2024-09-22T03:14:21.394 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:14:21.394 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:14:21.394 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.394 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:14:21.394 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:21.394 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.394 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.394 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:14:21.394 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:14:21.395 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.395 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.395 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:14:21.395 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:21.395 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.395 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.395 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:14:21.395 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:14:21.395 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.395 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.395 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:14:21.395 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:21.395 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.396 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.396 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:14:21.396 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:14:21.396 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.396 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.396 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:14:21.396 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:21.396 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.396 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.396 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:14:21.396 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:14:21.396 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.396 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.396 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:14:21.397 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:21.397 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.397 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.397 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:14:21.397 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:14:21.397 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.397 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.397 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:14:21.397 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:21.397 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.397 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.397 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:14:21.397 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:14:21.397 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.398 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.398 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:14:21.398 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:21.398 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.398 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.398 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:14:21.398 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:14:21.398 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.398 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.398 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:14:21.398 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:21.398 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.398 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.398 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:14:21.398 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:14:21.399 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.399 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.399 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:14:21.399 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:21.399 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.399 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.399 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:14:21.399 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:14:21.399 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.399 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.399 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:14:21.399 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:21.399 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.400 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.400 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:14:21.400 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:14:21.400 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.400 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.400 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:14:21.400 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:21.400 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.400 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.400 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:14:21.400 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:14:21.400 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.400 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.400 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:14:21.401 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:21.401 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.401 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.401 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:14:21.401 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:14:21.401 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.401 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.401 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:14:21.401 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:21.401 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.401 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.401 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:14:21.401 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:14:21.401 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.402 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:14:21.402 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:14:21.402 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.402 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-22T03:14:21.402 INFO:teuthology.orchestra.run.smithi196.stdout: "type": "client_request", 2024-09-22T03:14:21.402 INFO:teuthology.orchestra.run.smithi196.stdout: "id": 35, 2024-09-22T03:14:21.402 INFO:teuthology.orchestra.run.smithi196.stdout: "detail": { 2024-09-22T03:14:21.402 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:14:21.402 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:14:21.402 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:14:21.033318+0000" 2024-09-22T03:14:21.402 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.402 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:14:21.402 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:14:21.402 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:14:21.033323+0000" 2024-09-22T03:14:21.403 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.403 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:14:21.403 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:21.403 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.403 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.403 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:14:21.403 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:14:21.403 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:14:21.034375+0000" 2024-09-22T03:14:21.403 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.403 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:14:21.403 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:21.403 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.403 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.403 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:14:21.404 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:14:21.404 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.404 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.404 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:14:21.404 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:21.404 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.404 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.404 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:14:21.404 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:14:21.404 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:14:21.034891+0000" 2024-09-22T03:14:21.404 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.404 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:14:21.404 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:21.404 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.404 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.405 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:14:21.405 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:14:21.405 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:14:21.035189+0000", 2024-09-22T03:14:21.405 INFO:teuthology.orchestra.run.smithi196.stdout: "blocker": { 2024-09-22T03:14:21.405 INFO:teuthology.orchestra.run.smithi196.stdout: "op_type": "PerShardPipeline::create_or_wait_pg", 2024-09-22T03:14:21.405 INFO:teuthology.orchestra.run.smithi196.stdout: "detail": { 2024-09-22T03:14:21.405 INFO:teuthology.orchestra.run.smithi196.stdout: "waiting": 1, 2024-09-22T03:14:21.405 INFO:teuthology.orchestra.run.smithi196.stdout: "held_by_operation_id": 32 2024-09-22T03:14:21.405 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:14:21.405 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:14:21.405 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.405 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:14:21.405 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:21.405 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.406 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.406 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:14:21.406 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:14:21.406 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.406 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.406 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:14:21.406 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:21.406 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.406 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.406 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:14:21.406 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:14:21.406 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.406 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.406 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:14:21.406 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:14:21.407 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.407 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.407 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:14:21.407 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:21.407 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.407 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.407 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:14:21.407 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:14:21.407 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.407 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.407 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:14:21.407 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:21.407 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.407 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.407 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:14:21.407 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:14:21.408 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.408 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.408 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:14:21.408 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:21.408 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.408 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.408 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:14:21.408 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:14:21.408 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.408 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.408 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:14:21.408 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:21.408 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.408 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.409 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:14:21.409 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:14:21.409 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.409 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.409 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:14:21.409 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:21.409 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.409 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.409 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:14:21.409 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:14:21.409 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.409 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.409 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:14:21.409 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:21.409 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.410 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.410 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:14:21.410 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:14:21.410 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.410 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.410 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:14:21.410 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:21.410 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.410 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.410 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:14:21.410 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:14:21.410 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.410 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.410 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:14:21.410 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:21.411 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.411 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.411 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:14:21.411 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:14:21.411 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.411 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.411 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:14:21.411 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:21.411 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.411 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.411 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:14:21.411 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:14:21.411 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.411 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.411 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:14:21.411 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:21.412 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.412 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.412 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:14:21.412 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:14:21.412 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.412 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.412 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:14:21.412 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:21.412 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.412 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.412 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:14:21.412 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:14:21.412 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.412 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.412 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:14:21.413 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:21.413 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.413 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.413 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:14:21.413 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:14:21.413 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.413 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.413 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:14:21.413 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:21.413 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.413 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.413 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:14:21.413 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:14:21.413 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.413 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:14:21.413 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:14:21.414 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.414 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-22T03:14:21.414 INFO:teuthology.orchestra.run.smithi196.stdout: "type": "client_request", 2024-09-22T03:14:21.414 INFO:teuthology.orchestra.run.smithi196.stdout: "id": 36, 2024-09-22T03:14:21.414 INFO:teuthology.orchestra.run.smithi196.stdout: "detail": { 2024-09-22T03:14:21.414 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:14:21.414 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:14:21.414 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:14:21.035809+0000" 2024-09-22T03:14:21.414 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.414 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:14:21.414 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:14:21.414 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:14:21.035814+0000" 2024-09-22T03:14:21.414 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.414 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:14:21.415 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:21.415 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.415 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.415 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:14:21.415 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:14:21.415 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:14:21.035876+0000" 2024-09-22T03:14:21.415 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.415 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:14:21.415 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:21.415 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.415 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.415 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:14:21.415 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:14:21.415 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.415 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.416 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:14:21.416 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:21.416 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.416 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.416 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:14:21.416 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:14:21.416 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:14:21.036189+0000", 2024-09-22T03:14:21.416 INFO:teuthology.orchestra.run.smithi196.stdout: "blocker": { 2024-09-22T03:14:21.416 INFO:teuthology.orchestra.run.smithi196.stdout: "op_type": "ConnectionPipeline::get_pg_mapping", 2024-09-22T03:14:21.416 INFO:teuthology.orchestra.run.smithi196.stdout: "detail": { 2024-09-22T03:14:21.416 INFO:teuthology.orchestra.run.smithi196.stdout: "waiting": 1, 2024-09-22T03:14:21.416 INFO:teuthology.orchestra.run.smithi196.stdout: "held_by_operation_id": 35 2024-09-22T03:14:21.416 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:14:21.416 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:14:21.416 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.416 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:14:21.416 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:21.417 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.417 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.417 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:14:21.417 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:14:21.417 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.417 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.417 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:14:21.417 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:21.417 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.417 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.417 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:14:21.417 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:14:21.417 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.417 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.417 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:14:21.417 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:21.418 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.418 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.418 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:14:21.418 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:14:21.418 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.418 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.418 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:14:21.418 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:14:21.418 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.418 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.418 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:14:21.418 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:21.418 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.418 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.418 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:14:21.419 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:14:21.419 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.419 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.419 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:14:21.419 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:21.419 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.419 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.419 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:14:21.419 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:14:21.419 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.419 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.419 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:14:21.419 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:21.419 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.420 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.420 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:14:21.420 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:14:21.420 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.420 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.420 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:14:21.420 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:21.420 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.420 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.420 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:14:21.420 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:14:21.420 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.420 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.420 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:14:21.420 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:21.421 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.421 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.421 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:14:21.421 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:14:21.421 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.421 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.421 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:14:21.421 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:21.421 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.421 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.421 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:14:21.421 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:14:21.421 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.421 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.422 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:14:21.422 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:21.422 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.422 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.422 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:14:21.422 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:14:21.422 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.422 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.422 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:14:21.422 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:21.422 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.422 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.422 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:14:21.422 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:14:21.423 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.423 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.423 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:14:21.423 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:21.423 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.423 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.423 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:14:21.423 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:14:21.423 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.423 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.423 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:14:21.423 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:21.423 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.423 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.423 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:14:21.424 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:14:21.424 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.424 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.424 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:14:21.424 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:21.424 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.424 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.424 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:14:21.424 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:14:21.424 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.424 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.424 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:14:21.424 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:21.424 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.424 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.425 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:14:21.425 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:14:21.425 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.425 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.425 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:14:21.425 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:21.425 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.425 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.425 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:14:21.425 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:14:21.425 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.425 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:14:21.425 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:14:21.425 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.425 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-22T03:14:21.426 INFO:teuthology.orchestra.run.smithi196.stdout: "type": "client_request", 2024-09-22T03:14:21.426 INFO:teuthology.orchestra.run.smithi196.stdout: "id": 38, 2024-09-22T03:14:21.426 INFO:teuthology.orchestra.run.smithi196.stdout: "detail": { 2024-09-22T03:14:21.426 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:14:21.426 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:14:21.426 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:14:21.036663+0000" 2024-09-22T03:14:21.426 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.426 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:14:21.426 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:14:21.426 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:14:21.036667+0000" 2024-09-22T03:14:21.426 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.426 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:14:21.426 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:21.427 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.427 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.427 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:14:21.427 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:14:21.427 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:14:21.036783+0000", 2024-09-22T03:14:21.427 INFO:teuthology.orchestra.run.smithi196.stdout: "blocker": { 2024-09-22T03:14:21.427 INFO:teuthology.orchestra.run.smithi196.stdout: "op_type": "ConnectionPipeline::await_map", 2024-09-22T03:14:21.427 INFO:teuthology.orchestra.run.smithi196.stdout: "detail": { 2024-09-22T03:14:21.427 INFO:teuthology.orchestra.run.smithi196.stdout: "waiting": 1, 2024-09-22T03:14:21.427 INFO:teuthology.orchestra.run.smithi196.stdout: "held_by_operation_id": 36 2024-09-22T03:14:21.427 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:14:21.427 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:14:21.427 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.427 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:14:21.427 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:21.428 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.428 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.428 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:14:21.428 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:14:21.428 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.428 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.428 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:14:21.428 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:21.428 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.428 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.428 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:14:21.428 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:14:21.428 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.428 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.428 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:14:21.429 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:21.429 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.429 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.429 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:14:21.429 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:14:21.429 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.429 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.429 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:14:21.429 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:21.429 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.429 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.429 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:14:21.429 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:14:21.429 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.430 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.430 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:14:21.430 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:21.430 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.430 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.430 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:14:21.430 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:14:21.430 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.430 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.430 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:14:21.430 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:14:21.430 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.430 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.430 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:14:21.430 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:21.431 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.431 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.431 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:14:21.431 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:14:21.431 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.431 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.431 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:14:21.431 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:21.431 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.431 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.431 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:14:21.431 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:14:21.431 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.431 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.431 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:14:21.432 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:21.432 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.432 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.432 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:14:21.432 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:14:21.432 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.432 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.432 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:14:21.432 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:21.432 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.432 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.432 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:14:21.432 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:14:21.432 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.433 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.433 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:14:21.433 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:21.433 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.433 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.433 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:14:21.433 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:14:21.433 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.433 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.433 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:14:21.433 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:21.433 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.433 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.433 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:14:21.433 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:14:21.434 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.434 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.434 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:14:21.434 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:21.434 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.434 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.434 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:14:21.434 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:14:21.434 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.434 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.434 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:14:21.434 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:21.434 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.434 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.434 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:14:21.435 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:14:21.435 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.435 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.435 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:14:21.435 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:21.435 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.435 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.435 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:14:21.435 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:14:21.435 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.435 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.435 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:14:21.435 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:21.435 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.435 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.436 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:14:21.436 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:14:21.436 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.436 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.436 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:14:21.436 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:21.436 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.436 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.436 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:14:21.436 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:14:21.436 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.436 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.437 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:14:21.437 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:21.437 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.437 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.437 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:14:21.437 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:14:21.437 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.437 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.437 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:14:21.437 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:21.437 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.437 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.437 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:14:21.437 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:14:21.438 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.438 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:14:21.438 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:14:21.438 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.438 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-22T03:14:21.438 INFO:teuthology.orchestra.run.smithi196.stdout: "type": "client_request", 2024-09-22T03:14:21.438 INFO:teuthology.orchestra.run.smithi196.stdout: "id": 40, 2024-09-22T03:14:21.438 INFO:teuthology.orchestra.run.smithi196.stdout: "detail": { 2024-09-22T03:14:21.438 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:14:21.438 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:14:21.438 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:14:21.038054+0000" 2024-09-22T03:14:21.438 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.438 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:14:21.439 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:14:21.439 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:14:21.038087+0000", 2024-09-22T03:14:21.439 INFO:teuthology.orchestra.run.smithi196.stdout: "blocker": { 2024-09-22T03:14:21.439 INFO:teuthology.orchestra.run.smithi196.stdout: "op_type": "ConnectionPipeline::await_active", 2024-09-22T03:14:21.439 INFO:teuthology.orchestra.run.smithi196.stdout: "detail": { 2024-09-22T03:14:21.439 INFO:teuthology.orchestra.run.smithi196.stdout: "waiting": 2, 2024-09-22T03:14:21.439 INFO:teuthology.orchestra.run.smithi196.stdout: "held_by_operation_id": 38 2024-09-22T03:14:21.439 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:14:21.439 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:14:21.439 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.439 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:14:21.439 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:21.439 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.440 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.440 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:14:21.440 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:14:21.440 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.440 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.440 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:14:21.440 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:21.440 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.440 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.440 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:14:21.440 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:14:21.440 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.440 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.440 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:14:21.440 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:21.441 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.441 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.441 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:14:21.441 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:14:21.441 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.441 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.441 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:14:21.441 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:21.441 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.441 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.441 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:14:21.441 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:14:21.441 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.441 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.441 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:14:21.441 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:21.442 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.442 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.442 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:14:21.442 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:14:21.442 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.442 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.442 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:14:21.442 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:21.442 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.442 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.442 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:14:21.442 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:14:21.442 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.442 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.443 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:14:21.443 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:14:21.443 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.443 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.443 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:14:21.443 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:21.443 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.443 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.443 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:14:21.443 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:14:21.443 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.443 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.443 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:14:21.443 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:21.443 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.444 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.444 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:14:21.444 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:14:21.444 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.444 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.444 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:14:21.444 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:21.444 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.444 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.444 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:14:21.444 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:14:21.444 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.444 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.444 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:14:21.445 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:21.445 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.445 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.445 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:14:21.445 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:14:21.445 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.445 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.445 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:14:21.445 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:21.445 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.445 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.445 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:14:21.445 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:14:21.445 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.445 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.446 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:14:21.446 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:21.446 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.446 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.446 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:14:21.446 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:14:21.446 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.446 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.446 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:14:21.446 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:21.446 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.446 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.446 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:14:21.446 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:14:21.447 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.447 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.447 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:14:21.447 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:21.447 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.447 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.447 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:14:21.447 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:14:21.447 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.447 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.447 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:14:21.447 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:21.447 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.447 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.448 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:14:21.448 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:14:21.448 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.448 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.448 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:14:21.448 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:21.448 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.448 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.448 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:14:21.448 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:14:21.448 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.448 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.448 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:14:21.448 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:21.448 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.449 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.449 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:14:21.449 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:14:21.449 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.449 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.449 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:14:21.449 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:21.449 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.449 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.449 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:14:21.449 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:14:21.449 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.449 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.449 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:14:21.450 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:21.450 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.450 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.450 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:14:21.450 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:14:21.450 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.450 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:14:21.450 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:14:21.450 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.450 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-22T03:14:21.450 INFO:teuthology.orchestra.run.smithi196.stdout: "type": "client_request", 2024-09-22T03:14:21.450 INFO:teuthology.orchestra.run.smithi196.stdout: "id": 41, 2024-09-22T03:14:21.450 INFO:teuthology.orchestra.run.smithi196.stdout: "detail": { 2024-09-22T03:14:21.450 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:14:21.451 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:14:21.451 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:14:21.038451+0000" 2024-09-22T03:14:21.451 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.451 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:14:21.451 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:14:21.451 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:14:21.038456+0000", 2024-09-22T03:14:21.451 INFO:teuthology.orchestra.run.smithi196.stdout: "blocker": { 2024-09-22T03:14:21.451 INFO:teuthology.orchestra.run.smithi196.stdout: "op_type": "ConnectionPipeline::await_active", 2024-09-22T03:14:21.451 INFO:teuthology.orchestra.run.smithi196.stdout: "detail": { 2024-09-22T03:14:21.451 INFO:teuthology.orchestra.run.smithi196.stdout: "waiting": 2, 2024-09-22T03:14:21.451 INFO:teuthology.orchestra.run.smithi196.stdout: "held_by_operation_id": 38 2024-09-22T03:14:21.451 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:14:21.451 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:14:21.451 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.452 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:14:21.452 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:21.452 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.452 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.452 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:14:21.452 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:14:21.452 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.452 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.452 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:14:21.452 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:21.452 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.452 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.452 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:14:21.453 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:14:21.453 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.453 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.453 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:14:21.453 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:21.453 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.453 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.453 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:14:21.453 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:14:21.453 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.453 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.453 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:14:21.453 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:21.453 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.453 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.453 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:14:21.454 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:14:21.454 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.454 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.454 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:14:21.454 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:21.454 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.454 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.454 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:14:21.454 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:14:21.454 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.454 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.454 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:14:21.454 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:21.454 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.454 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.455 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:14:21.455 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:14:21.455 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.455 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.455 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:14:21.455 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:14:21.455 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.455 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.455 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:14:21.455 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:21.455 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.455 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.455 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:14:21.455 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:14:21.456 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.456 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.456 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:14:21.456 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:21.456 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.456 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.456 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:14:21.456 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:14:21.456 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.456 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.456 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:14:21.456 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:21.456 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.456 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.456 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:14:21.457 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:14:21.457 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.457 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.457 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:14:21.457 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:21.457 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.457 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.457 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:14:21.457 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:14:21.457 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.457 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.457 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:14:21.457 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:21.457 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.457 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.457 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:14:21.457 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:14:21.458 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.458 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.458 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:14:21.458 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:21.458 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.458 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.458 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:14:21.458 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:14:21.458 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.458 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.458 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:14:21.458 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:21.458 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.458 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.458 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:14:21.459 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:14:21.459 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.459 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.459 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:14:21.459 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:21.459 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.459 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.459 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:14:21.459 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:14:21.459 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.459 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.459 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:14:21.459 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:21.459 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.459 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.460 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:14:21.460 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:14:21.460 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.460 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.460 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:14:21.460 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:21.460 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.460 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.460 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:14:21.460 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:14:21.460 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.460 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.460 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:14:21.460 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:21.461 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.461 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.461 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:14:21.461 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:14:21.461 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.461 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.461 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:14:21.461 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:21.461 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.461 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.461 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:14:21.461 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:14:21.461 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.461 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.461 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:14:21.462 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:21.462 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.462 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:14:21.462 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:14:21.462 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:14:21.462 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:21.462 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:14:21.462 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:14:21.462 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:14:21.462 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-22T03:14:21.462 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-09-22T03:14:22.012 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 42 16 41 25 0.00464971 0 - 9.72339 2024-09-22T03:14:23.012 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 43 16 391 375 0.0681238 0.182292 0.0318378 1.83398 2024-09-22T03:14:23.102 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-22T03:14:23.231 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:14:23.228+0000 7f529ff17640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:14:23.234 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:14:23.233+0000 7f529ff17640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:14:23.238 INFO:teuthology.orchestra.run.smithi153.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-22T03:14:23.365 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-22T03:14:23.366 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-22T03:14:24.012 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 44 16 754 738 0.131021 2.83594 0.0177605 0.953258 2024-09-22T03:14:25.013 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 45 16 1112 1096 0.190254 2.79688 0.0260383 0.656852 2024-09-22T03:14:25.367 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-22T03:14:25.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:14:25.481+0000 7fcd04757640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:14:25.487 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:14:25.486+0000 7fcd04757640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:14:25.491 INFO:teuthology.orchestra.run.smithi153.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-22T03:14:25.597 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-22T03:14:25.597 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-22T03:14:26.013 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 46 16 1490 1474 0.250309 2.95312 0.0279328 0.499217 2024-09-22T03:14:26.464 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [0] live_osds: [1, 3, 2] 2024-09-22T03:14:26.464 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-22T03:14:26.464 INFO:tasks.thrashosds.thrasher:Reviving osd 0 2024-09-22T03:14:26.464 INFO:tasks.ceph.osd.0:Restarting daemon 2024-09-22T03:14:26.464 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2>> /var/log/ceph/ceph-osd.0.log 2024-09-22T03:14:26.467 INFO:tasks.ceph.osd.0:Started 2024-09-22T03:14:26.467 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-22T03:14:26.576 INFO:teuthology.orchestra.run.smithi153.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-09-22T03:14:26.580 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-22T03:14:26.581 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-0, ['dump_ops_in_flight'] 2024-09-22T03:14:27.013 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 47 16 1841 1825 0.303321 2.74219 0.0896417 0.411832 2024-09-22T03:14:27.598 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-22T03:14:27.708 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:14:27.705+0000 7f5ade197640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:14:27.712 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:14:27.710+0000 7f5ade197640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:14:27.717 INFO:teuthology.orchestra.run.smithi153.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-22T03:14:27.832 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-22T03:14:27.832 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-22T03:14:28.013 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 48 16 2203 2187 0.355915 2.82812 0.0676386 0.35107 2024-09-22T03:14:29.013 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 49 16 2555 2539 0.404767 2.75 0.0471737 0.308712 2024-09-22T03:14:29.833 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-22T03:14:29.950 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:14:29.946+0000 7fc0a98d7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:14:29.952 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:14:29.951+0000 7fc0a98d7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:14:29.957 INFO:teuthology.orchestra.run.smithi153.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-22T03:14:30.013 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 50 16 2899 2883 0.450416 2.6875 0.0665308 0.277313 2024-09-22T03:14:30.031 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-22T03:14:30.031 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-22T03:14:31.013 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 51 16 3186 3170 0.485544 2.24219 0.0260114 0.257378 2024-09-22T03:14:31.582 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-22T03:14:31.684 INFO:teuthology.orchestra.run.smithi153.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-09-22T03:14:31.686 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-22T03:14:31.687 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-0, ['dump_ops_in_flight'] 2024-09-22T03:14:32.013 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 52 16 3480 3464 0.520373 2.29688 0.0569407 0.240039 2024-09-22T03:14:32.033 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-22T03:14:32.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:14:32.138+0000 7f8c75ece640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:14:32.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:14:32.143+0000 7f8c75ece640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:14:32.150 INFO:teuthology.orchestra.run.smithi153.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-22T03:14:32.402 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-22T03:14:32.403 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-22T03:14:33.013 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 53 16 3796 3780 0.557129 2.46875 0.0583057 0.224239 2024-09-22T03:14:34.013 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 54 16 4121 4105 0.593827 2.53906 0.0359674 0.210344 2024-09-22T03:14:34.403 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-22T03:14:34.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:14:34.529+0000 7f3966797640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:14:34.537 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:14:34.536+0000 7f3966797640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:14:34.544 INFO:teuthology.orchestra.run.smithi153.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-22T03:14:34.747 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-22T03:14:34.747 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-22T03:14:35.013 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 55 16 4465 4449 0.631888 2.6875 0.0647425 0.19769 2024-09-22T03:14:36.014 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 56 16 4830 4814 0.67152 2.85156 0.0198208 0.186097 2024-09-22T03:14:36.688 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-22T03:14:36.748 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-22T03:14:36.804 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-22T03:14:36.805 INFO:teuthology.orchestra.run.smithi153.stdout: "ops_in_flight": [ 2024-09-22T03:14:36.805 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-22T03:14:36.805 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "client_request", 2024-09-22T03:14:36.805 INFO:teuthology.orchestra.run.smithi153.stdout: "id": 56, 2024-09-22T03:14:36.805 INFO:teuthology.orchestra.run.smithi153.stdout: "detail": { 2024-09-22T03:14:36.805 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:14:36.805 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:14:36.805 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:14:36.367797+0000" 2024-09-22T03:14:36.805 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.805 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:14:36.805 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:14:36.805 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:14:36.367819+0000" 2024-09-22T03:14:36.805 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.805 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:14:36.805 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:36.806 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.806 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.806 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:14:36.806 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:14:36.806 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:14:36.368900+0000" 2024-09-22T03:14:36.806 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.806 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:14:36.806 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:36.806 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.806 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.806 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:14:36.806 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:14:36.806 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:14:36.369147+0000" 2024-09-22T03:14:36.806 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.806 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:14:36.807 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:36.807 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.807 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.807 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:14:36.807 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:14:36.807 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:14:36.421478+0000" 2024-09-22T03:14:36.807 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.807 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:14:36.807 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:36.807 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.807 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.807 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:14:36.807 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:14:36.807 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:14:36.425090+0000" 2024-09-22T03:14:36.807 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.808 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:14:36.808 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:36.808 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.808 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.808 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:14:36.808 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:14:36.808 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:14:36.425220+0000" 2024-09-22T03:14:36.808 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.808 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:14:36.808 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:36.808 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.808 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.808 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:14:36.808 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:14:36.808 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.809 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.809 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:14:36.809 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:14:36.809 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:14:36.499769+0000" 2024-09-22T03:14:36.809 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.809 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:14:36.809 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:36.809 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.809 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.809 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:14:36.809 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:14:36.809 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.809 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.809 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:14:36.809 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:36.810 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.810 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.810 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:14:36.810 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:14:36.810 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:14:36.499985+0000" 2024-09-22T03:14:36.810 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.810 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:14:36.810 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:36.810 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.810 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.810 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:14:36.811 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:14:36.811 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:14:36.500434+0000", 2024-09-22T03:14:36.811 INFO:teuthology.orchestra.run.smithi153.stdout: "blocker": { 2024-09-22T03:14:36.811 INFO:teuthology.orchestra.run.smithi153.stdout: "op_type": "PGActivationBlocker", 2024-09-22T03:14:36.811 INFO:teuthology.orchestra.run.smithi153.stdout: "detail": { 2024-09-22T03:14:36.811 INFO:teuthology.orchestra.run.smithi153.stdout: "pgid": "6.7" 2024-09-22T03:14:36.811 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:14:36.811 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:14:36.811 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.811 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:14:36.811 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:36.811 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.811 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.811 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:14:36.812 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:14:36.812 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.812 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.812 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:14:36.812 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:36.812 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.812 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.812 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:14:36.812 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:14:36.812 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.812 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.812 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:14:36.812 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:36.812 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.813 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.813 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:14:36.813 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:14:36.813 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.813 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.813 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:14:36.813 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:36.813 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.813 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.813 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:14:36.813 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:14:36.813 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.813 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.813 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:14:36.813 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:36.814 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.814 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.814 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:14:36.814 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:14:36.814 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.814 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.814 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:14:36.814 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:36.814 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.814 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.814 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:14:36.815 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:14:36.815 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.815 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.815 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:14:36.815 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:36.815 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.815 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.815 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:14:36.815 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:14:36.815 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.815 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.815 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:14:36.815 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:36.815 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.815 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.816 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:14:36.816 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:14:36.816 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.816 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.816 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:14:36.816 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:36.816 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.816 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.816 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:14:36.816 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:14:36.816 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.816 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.816 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:14:36.816 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:36.816 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.817 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.817 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:14:36.817 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:14:36.817 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.817 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:14:36.817 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:14:36.817 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.817 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-22T03:14:36.817 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "client_request", 2024-09-22T03:14:36.817 INFO:teuthology.orchestra.run.smithi153.stdout: "id": 58, 2024-09-22T03:14:36.817 INFO:teuthology.orchestra.run.smithi153.stdout: "detail": { 2024-09-22T03:14:36.817 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:14:36.817 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:14:36.818 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:14:36.376781+0000" 2024-09-22T03:14:36.818 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.818 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:14:36.818 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:14:36.818 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:14:36.376788+0000" 2024-09-22T03:14:36.818 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.818 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:14:36.818 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:36.818 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.818 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.818 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:14:36.818 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:14:36.818 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:14:36.377742+0000" 2024-09-22T03:14:36.818 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.819 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:14:36.819 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:36.819 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.819 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.819 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:14:36.819 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:14:36.819 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.819 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.819 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:14:36.819 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:36.819 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.819 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.819 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:14:36.819 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:14:36.819 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:14:36.422540+0000" 2024-09-22T03:14:36.820 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.820 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:14:36.820 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:36.820 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.820 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.820 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:14:36.820 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:14:36.820 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:14:36.426114+0000" 2024-09-22T03:14:36.820 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.820 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:14:36.820 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:36.820 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.820 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.820 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:14:36.820 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:14:36.821 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.821 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.821 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:14:36.821 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:36.821 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.821 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.821 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:14:36.821 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:14:36.821 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.821 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.821 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:14:36.821 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:14:36.821 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:14:36.500302+0000" 2024-09-22T03:14:36.821 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.821 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:14:36.822 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:36.822 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.822 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.822 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:14:36.822 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:14:36.822 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.822 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.822 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:14:36.822 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:36.822 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.822 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.822 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:14:36.822 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:14:36.822 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:14:36.500688+0000", 2024-09-22T03:14:36.822 INFO:teuthology.orchestra.run.smithi153.stdout: "blocker": { 2024-09-22T03:14:36.822 INFO:teuthology.orchestra.run.smithi153.stdout: "op_type": "CommonPGPipeline:::wait_for_active", 2024-09-22T03:14:36.823 INFO:teuthology.orchestra.run.smithi153.stdout: "detail": { 2024-09-22T03:14:36.823 INFO:teuthology.orchestra.run.smithi153.stdout: "waiting": 1, 2024-09-22T03:14:36.823 INFO:teuthology.orchestra.run.smithi153.stdout: "held_by_operation_id": 56 2024-09-22T03:14:36.823 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:14:36.823 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:14:36.823 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.823 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:14:36.823 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:36.823 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.823 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.823 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:14:36.823 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:14:36.823 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.823 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.823 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:14:36.824 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:36.824 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.824 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.824 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:14:36.824 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:14:36.824 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.824 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.824 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:14:36.824 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:36.824 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.824 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.824 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:14:36.824 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:14:36.824 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.824 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.825 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:14:36.825 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:36.825 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.825 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.825 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:14:36.825 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:14:36.825 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.825 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.825 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:14:36.825 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:36.825 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.825 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.825 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:14:36.825 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:14:36.825 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.826 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.826 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:14:36.826 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:36.826 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.826 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.826 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:14:36.826 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:14:36.826 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.826 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.826 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:14:36.826 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:36.826 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.826 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.826 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:14:36.827 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:14:36.827 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.827 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.827 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:14:36.827 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:36.827 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.827 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.827 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:14:36.827 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:14:36.827 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.827 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.827 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:14:36.827 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:36.827 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.827 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.828 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:14:36.828 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:14:36.828 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.828 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.828 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:14:36.828 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:36.828 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.828 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.828 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:14:36.828 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:14:36.828 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.828 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.828 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:14:36.828 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:36.829 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.829 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.829 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:14:36.829 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:14:36.829 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.829 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:14:36.829 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:14:36.829 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.829 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-22T03:14:36.829 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "client_request", 2024-09-22T03:14:36.829 INFO:teuthology.orchestra.run.smithi153.stdout: "id": 59, 2024-09-22T03:14:36.829 INFO:teuthology.orchestra.run.smithi153.stdout: "detail": { 2024-09-22T03:14:36.829 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:14:36.829 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:14:36.830 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:14:36.380407+0000" 2024-09-22T03:14:36.830 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.830 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:14:36.830 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:14:36.830 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:14:36.380419+0000" 2024-09-22T03:14:36.830 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.830 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:14:36.830 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:36.830 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.830 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.830 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:14:36.830 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:14:36.831 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:14:36.421784+0000" 2024-09-22T03:14:36.831 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.831 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:14:36.831 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:36.831 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.831 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.831 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:14:36.831 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:14:36.831 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.831 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.831 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:14:36.831 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:36.831 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.831 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.832 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:14:36.832 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:14:36.832 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:14:36.426013+0000" 2024-09-22T03:14:36.832 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.832 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:14:36.832 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:36.832 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.832 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.832 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:14:36.832 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:14:36.832 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:14:36.428836+0000" 2024-09-22T03:14:36.832 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.832 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:14:36.832 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:36.832 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.832 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.833 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:14:36.833 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:14:36.833 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.833 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.833 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:14:36.833 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:36.833 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.833 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.833 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:14:36.833 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:14:36.833 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.833 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.833 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:14:36.833 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:14:36.833 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:14:36.500927+0000", 2024-09-22T03:14:36.834 INFO:teuthology.orchestra.run.smithi153.stdout: "blocker": { 2024-09-22T03:14:36.834 INFO:teuthology.orchestra.run.smithi153.stdout: "op_type": "ClientRequest::PGPipeline::await_map", 2024-09-22T03:14:36.834 INFO:teuthology.orchestra.run.smithi153.stdout: "detail": { 2024-09-22T03:14:36.834 INFO:teuthology.orchestra.run.smithi153.stdout: "waiting": 1, 2024-09-22T03:14:36.834 INFO:teuthology.orchestra.run.smithi153.stdout: "held_by_operation_id": 58 2024-09-22T03:14:36.834 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:14:36.834 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:14:36.834 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.834 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:14:36.834 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:36.834 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.834 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.834 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:14:36.834 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:14:36.834 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.835 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.835 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:14:36.835 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:36.835 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.835 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.835 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:14:36.835 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:14:36.835 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.835 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.835 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:14:36.835 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:36.835 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.835 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.835 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:14:36.835 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:14:36.835 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.836 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.836 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:14:36.836 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:36.836 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.836 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.836 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:14:36.836 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:14:36.836 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.836 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.836 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:14:36.836 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:36.836 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.836 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.836 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:14:36.837 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:14:36.837 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.837 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.837 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:14:36.837 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:36.837 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.837 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.837 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:14:36.837 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:14:36.837 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.837 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.837 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:14:36.837 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:36.837 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.838 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.838 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:14:36.838 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:14:36.838 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.838 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.838 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:14:36.838 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:36.838 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.838 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.838 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:14:36.838 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:14:36.838 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.838 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.838 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:14:36.838 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:36.839 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.839 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.839 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:14:36.839 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:14:36.839 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.839 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.839 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:14:36.839 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:36.839 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.839 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.839 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:14:36.839 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:14:36.839 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.839 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.839 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:14:36.839 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:36.840 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.840 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.840 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:14:36.840 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:14:36.840 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.840 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.840 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:14:36.840 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:36.840 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.840 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.840 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:14:36.840 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:14:36.840 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.840 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.840 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:14:36.841 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:36.841 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.841 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.841 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:14:36.841 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:14:36.841 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.841 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:14:36.841 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:14:36.841 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.841 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-22T03:14:36.841 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "client_request", 2024-09-22T03:14:36.841 INFO:teuthology.orchestra.run.smithi153.stdout: "id": 62, 2024-09-22T03:14:36.841 INFO:teuthology.orchestra.run.smithi153.stdout: "detail": { 2024-09-22T03:14:36.841 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:14:36.842 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:14:36.842 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:14:36.384503+0000" 2024-09-22T03:14:36.842 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.842 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:14:36.842 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:14:36.842 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:14:36.384508+0000" 2024-09-22T03:14:36.842 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.842 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:14:36.842 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:36.842 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.842 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.842 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:14:36.842 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:14:36.842 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:14:36.427054+0000" 2024-09-22T03:14:36.842 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.842 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:14:36.843 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:36.843 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.843 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.843 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:14:36.843 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:14:36.843 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.843 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.843 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:14:36.843 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:36.843 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.843 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.843 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:14:36.843 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:14:36.843 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:14:36.427947+0000" 2024-09-22T03:14:36.843 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.844 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:14:36.844 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:36.844 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.844 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.844 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:14:36.844 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:14:36.844 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:14:36.429486+0000", 2024-09-22T03:14:36.844 INFO:teuthology.orchestra.run.smithi153.stdout: "blocker": { 2024-09-22T03:14:36.844 INFO:teuthology.orchestra.run.smithi153.stdout: "op_type": "PerShardPipeline::create_or_wait_pg", 2024-09-22T03:14:36.844 INFO:teuthology.orchestra.run.smithi153.stdout: "detail": { 2024-09-22T03:14:36.844 INFO:teuthology.orchestra.run.smithi153.stdout: "waiting": 5, 2024-09-22T03:14:36.844 INFO:teuthology.orchestra.run.smithi153.stdout: "held_by_operation_id": 59 2024-09-22T03:14:36.844 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:14:36.844 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:14:36.845 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.845 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:14:36.845 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:36.845 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.845 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.845 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:14:36.845 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:14:36.845 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.845 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.845 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:14:36.845 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:36.845 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.845 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.845 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:14:36.845 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:14:36.845 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.845 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.846 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:14:36.846 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:14:36.846 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.846 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.846 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:14:36.846 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:36.846 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.846 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.846 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:14:36.846 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:14:36.846 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.847 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.847 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:14:36.847 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:36.847 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.847 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.847 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:14:36.847 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:14:36.847 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.847 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.847 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:14:36.847 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:36.847 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.847 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.847 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:14:36.847 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:14:36.848 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.848 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.848 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:14:36.848 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:36.848 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.848 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.848 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:14:36.848 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:14:36.848 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.848 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.848 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:14:36.848 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:36.848 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.848 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.848 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:14:36.849 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:14:36.849 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.849 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.849 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:14:36.849 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:36.849 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.849 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.849 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:14:36.849 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:14:36.849 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.849 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.849 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:14:36.849 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:36.849 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.849 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.850 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:14:36.850 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:14:36.850 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.850 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.850 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:14:36.850 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:36.850 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.850 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.850 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:14:36.850 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:14:36.850 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.850 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.850 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:14:36.850 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:36.850 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.851 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.851 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:14:36.851 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:14:36.851 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.851 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.851 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:14:36.851 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:36.851 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.851 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.851 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:14:36.851 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:14:36.851 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.851 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.851 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:14:36.851 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:36.852 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.852 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.852 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:14:36.852 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:14:36.852 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.852 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.852 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:14:36.852 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:36.852 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.852 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.852 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:14:36.852 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:14:36.852 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.852 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.852 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:14:36.853 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:36.853 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.853 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.853 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:14:36.853 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:14:36.853 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.853 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:14:36.853 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:14:36.853 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.853 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-22T03:14:36.853 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "client_request", 2024-09-22T03:14:36.853 INFO:teuthology.orchestra.run.smithi153.stdout: "id": 64, 2024-09-22T03:14:36.853 INFO:teuthology.orchestra.run.smithi153.stdout: "detail": { 2024-09-22T03:14:36.854 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:14:36.854 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:14:36.854 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:14:36.388274+0000" 2024-09-22T03:14:36.854 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.854 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:14:36.854 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:14:36.854 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:14:36.388279+0000" 2024-09-22T03:14:36.854 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.854 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:14:36.854 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:36.854 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.854 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.854 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:14:36.855 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:14:36.855 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:14:36.428071+0000" 2024-09-22T03:14:36.855 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.855 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:14:36.855 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:36.855 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.855 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.855 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:14:36.855 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:14:36.855 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.855 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.855 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:14:36.855 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:36.855 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.855 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.855 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:14:36.856 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:14:36.856 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:14:36.429081+0000" 2024-09-22T03:14:36.856 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.856 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:14:36.856 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:36.856 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.856 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.856 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:14:36.856 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:14:36.856 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:14:36.431465+0000", 2024-09-22T03:14:36.856 INFO:teuthology.orchestra.run.smithi153.stdout: "blocker": { 2024-09-22T03:14:36.856 INFO:teuthology.orchestra.run.smithi153.stdout: "op_type": "PerShardPipeline::create_or_wait_pg", 2024-09-22T03:14:36.856 INFO:teuthology.orchestra.run.smithi153.stdout: "detail": { 2024-09-22T03:14:36.856 INFO:teuthology.orchestra.run.smithi153.stdout: "waiting": 5, 2024-09-22T03:14:36.857 INFO:teuthology.orchestra.run.smithi153.stdout: "held_by_operation_id": 59 2024-09-22T03:14:36.857 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:14:36.857 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:14:36.857 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.857 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:14:36.857 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:36.857 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.857 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.857 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:14:36.857 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:14:36.858 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.858 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.858 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:14:36.858 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:36.858 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.858 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.859 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:14:36.859 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:14:36.859 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.859 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.859 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:14:36.859 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:14:36.859 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.860 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.860 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:14:36.860 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:36.860 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.860 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.860 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:14:36.861 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:14:36.861 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.861 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.861 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:14:36.861 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:36.861 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.861 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.862 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:14:36.862 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:14:36.862 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.862 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.862 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:14:36.862 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:36.862 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.862 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.862 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:14:36.862 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:14:36.862 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.862 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.862 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:14:36.862 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:36.862 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.863 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.863 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:14:36.863 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:14:36.863 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.863 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.863 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:14:36.863 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:36.863 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.863 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.863 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:14:36.863 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:14:36.863 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.863 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.863 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:14:36.863 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:36.863 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.864 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.864 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:14:36.864 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:14:36.864 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.864 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.864 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:14:36.864 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:36.864 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.864 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.864 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:14:36.864 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:14:36.864 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.864 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.865 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:14:36.865 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:36.865 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.865 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.865 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:14:36.865 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:14:36.865 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.865 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.865 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:14:36.865 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:36.865 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.865 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.866 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:14:36.866 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:14:36.866 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.866 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.866 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:14:36.866 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:36.866 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.866 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.866 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:14:36.866 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:14:36.866 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.866 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.866 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:14:36.866 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:36.867 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.867 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.867 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:14:36.867 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:14:36.867 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.867 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.867 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:14:36.867 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:36.867 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.867 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.867 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:14:36.867 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:14:36.867 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.867 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.867 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:14:36.868 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:36.868 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.868 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.868 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:14:36.868 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:14:36.868 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.868 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:14:36.868 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:14:36.868 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.868 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-22T03:14:36.868 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "client_request", 2024-09-22T03:14:36.868 INFO:teuthology.orchestra.run.smithi153.stdout: "id": 68, 2024-09-22T03:14:36.868 INFO:teuthology.orchestra.run.smithi153.stdout: "detail": { 2024-09-22T03:14:36.868 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:14:36.868 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:14:36.868 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:14:36.392239+0000" 2024-09-22T03:14:36.868 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.869 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:14:36.869 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:14:36.869 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:14:36.392244+0000" 2024-09-22T03:14:36.869 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.869 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:14:36.869 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:36.869 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.869 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.869 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:14:36.869 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:14:36.869 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:14:36.429396+0000" 2024-09-22T03:14:36.869 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.869 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:14:36.869 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:36.869 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.870 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.870 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:14:36.870 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:14:36.870 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.870 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.870 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:14:36.870 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:36.870 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.870 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.870 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:14:36.870 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:14:36.870 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:14:36.430485+0000" 2024-09-22T03:14:36.870 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.870 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:14:36.870 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:36.870 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.871 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.871 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:14:36.871 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:14:36.871 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:14:36.432623+0000", 2024-09-22T03:14:36.871 INFO:teuthology.orchestra.run.smithi153.stdout: "blocker": { 2024-09-22T03:14:36.871 INFO:teuthology.orchestra.run.smithi153.stdout: "op_type": "PerShardPipeline::create_or_wait_pg", 2024-09-22T03:14:36.871 INFO:teuthology.orchestra.run.smithi153.stdout: "detail": { 2024-09-22T03:14:36.871 INFO:teuthology.orchestra.run.smithi153.stdout: "waiting": 5, 2024-09-22T03:14:36.871 INFO:teuthology.orchestra.run.smithi153.stdout: "held_by_operation_id": 59 2024-09-22T03:14:36.871 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:14:36.871 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:14:36.871 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.871 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:14:36.871 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:36.871 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.871 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.872 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:14:36.872 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:14:36.872 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.872 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.872 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:14:36.872 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:36.872 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.872 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.872 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:14:36.872 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:14:36.872 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.873 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.873 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:14:36.873 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:14:36.873 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.873 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.873 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:14:36.873 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:36.873 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.873 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.873 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:14:36.873 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:14:36.873 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.873 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.873 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:14:36.873 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:36.873 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.873 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.874 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:14:36.874 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:14:36.874 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.874 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.874 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:14:36.874 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:36.874 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.874 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.874 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:14:36.874 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:14:36.874 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.874 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.874 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:14:36.874 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:36.875 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.875 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.875 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:14:36.875 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:14:36.875 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.875 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.875 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:14:36.875 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:36.875 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.875 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.875 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:14:36.875 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:14:36.875 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.875 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.875 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:14:36.875 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:36.876 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.876 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.876 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:14:36.876 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:14:36.876 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.876 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.876 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:14:36.876 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:36.876 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.876 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.876 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:14:36.876 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:14:36.876 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.876 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.876 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:14:36.877 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:36.877 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.877 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.877 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:14:36.877 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:14:36.877 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.877 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.877 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:14:36.877 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:36.877 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.877 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.877 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:14:36.877 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:14:36.877 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.878 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.878 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:14:36.878 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:36.878 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.878 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.878 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:14:36.878 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:14:36.878 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.878 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.878 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:14:36.878 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:36.878 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.878 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.878 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:14:36.878 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:14:36.878 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.879 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.879 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:14:36.879 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:36.879 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.879 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.879 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:14:36.879 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:14:36.879 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.879 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.879 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:14:36.879 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:36.879 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.879 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.879 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:14:36.879 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:14:36.880 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.880 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:14:36.880 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:14:36.880 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.880 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-22T03:14:36.880 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "client_request", 2024-09-22T03:14:36.880 INFO:teuthology.orchestra.run.smithi153.stdout: "id": 71, 2024-09-22T03:14:36.880 INFO:teuthology.orchestra.run.smithi153.stdout: "detail": { 2024-09-22T03:14:36.880 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:14:36.880 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:14:36.880 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:14:36.396022+0000" 2024-09-22T03:14:36.880 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.880 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:14:36.880 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:14:36.880 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:14:36.396026+0000" 2024-09-22T03:14:36.880 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.881 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:14:36.881 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:36.881 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.881 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.881 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:14:36.881 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:14:36.881 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:14:36.430530+0000" 2024-09-22T03:14:36.881 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.881 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:14:36.881 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:36.881 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.881 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.881 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:14:36.881 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:14:36.882 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.882 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.882 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:14:36.882 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:36.882 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.882 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.882 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:14:36.882 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:14:36.882 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:14:36.432263+0000" 2024-09-22T03:14:36.882 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.882 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:14:36.882 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:36.882 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.882 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.883 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:14:36.883 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:14:36.883 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:14:36.433737+0000", 2024-09-22T03:14:36.883 INFO:teuthology.orchestra.run.smithi153.stdout: "blocker": { 2024-09-22T03:14:36.883 INFO:teuthology.orchestra.run.smithi153.stdout: "op_type": "PerShardPipeline::create_or_wait_pg", 2024-09-22T03:14:36.883 INFO:teuthology.orchestra.run.smithi153.stdout: "detail": { 2024-09-22T03:14:36.883 INFO:teuthology.orchestra.run.smithi153.stdout: "waiting": 5, 2024-09-22T03:14:36.883 INFO:teuthology.orchestra.run.smithi153.stdout: "held_by_operation_id": 59 2024-09-22T03:14:36.883 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:14:36.883 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:14:36.883 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.883 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:14:36.883 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:36.883 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.883 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.884 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:14:36.884 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:14:36.884 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.884 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.884 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:14:36.884 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:36.884 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.884 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.884 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:14:36.884 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:14:36.884 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.884 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.884 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:14:36.884 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:14:36.885 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.885 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.885 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:14:36.885 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:36.885 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.885 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.885 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:14:36.885 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:14:36.885 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.885 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.885 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:14:36.885 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:36.885 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.885 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.886 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:14:36.886 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:14:36.886 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.886 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.886 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:14:36.886 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:36.886 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.886 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.886 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:14:36.886 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:14:36.886 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.886 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.886 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:14:36.887 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:36.887 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.887 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.887 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:14:36.887 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:14:36.887 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.887 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.887 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:14:36.887 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:36.887 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.887 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.887 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:14:36.887 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:14:36.887 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.887 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.888 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:14:36.888 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:36.888 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.888 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.888 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:14:36.888 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:14:36.888 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.888 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.888 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:14:36.888 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:36.888 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.888 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.888 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:14:36.889 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:14:36.889 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.889 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.889 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:14:36.889 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:36.889 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.889 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.889 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:14:36.889 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:14:36.889 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.889 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.889 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:14:36.889 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:36.889 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.889 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.890 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:14:36.890 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:14:36.890 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.890 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.890 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:14:36.890 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:36.890 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.890 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.890 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:14:36.890 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:14:36.890 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.890 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.891 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:14:36.891 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:36.891 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.891 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.891 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:14:36.891 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:14:36.891 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.891 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.891 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:14:36.891 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:36.891 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.891 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.891 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:14:36.891 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:14:36.891 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.891 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.892 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:14:36.892 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:36.892 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.892 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.892 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:14:36.892 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:14:36.892 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.892 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:14:36.892 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:14:36.892 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.892 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-22T03:14:36.892 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "client_request", 2024-09-22T03:14:36.893 INFO:teuthology.orchestra.run.smithi153.stdout: "id": 77, 2024-09-22T03:14:36.893 INFO:teuthology.orchestra.run.smithi153.stdout: "detail": { 2024-09-22T03:14:36.893 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:14:36.893 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:14:36.893 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:14:36.403326+0000" 2024-09-22T03:14:36.893 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.893 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:14:36.893 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:14:36.893 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:14:36.403332+0000" 2024-09-22T03:14:36.893 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.893 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:14:36.893 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:36.893 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.893 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.893 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:14:36.893 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:14:36.894 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:14:36.432379+0000" 2024-09-22T03:14:36.894 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.894 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:14:36.894 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:36.894 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.894 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.894 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:14:36.894 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:14:36.894 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.894 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.894 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:14:36.894 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:36.894 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.894 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.894 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:14:36.895 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:14:36.895 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:14:36.433308+0000" 2024-09-22T03:14:36.895 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.895 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:14:36.895 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:36.895 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.895 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.895 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:14:36.895 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:14:36.895 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:14:36.433772+0000", 2024-09-22T03:14:36.895 INFO:teuthology.orchestra.run.smithi153.stdout: "blocker": { 2024-09-22T03:14:36.895 INFO:teuthology.orchestra.run.smithi153.stdout: "op_type": "PerShardPipeline::create_or_wait_pg", 2024-09-22T03:14:36.895 INFO:teuthology.orchestra.run.smithi153.stdout: "detail": { 2024-09-22T03:14:36.895 INFO:teuthology.orchestra.run.smithi153.stdout: "waiting": 5, 2024-09-22T03:14:36.895 INFO:teuthology.orchestra.run.smithi153.stdout: "held_by_operation_id": 59 2024-09-22T03:14:36.895 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:14:36.896 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:14:36.896 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.896 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:14:36.896 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:36.896 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.896 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.896 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:14:36.896 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:14:36.896 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.896 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.896 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:14:36.896 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:36.896 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.897 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.897 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:14:36.897 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:14:36.897 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.897 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.897 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:14:36.897 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:14:36.897 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.897 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.897 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:14:36.897 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:36.897 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.897 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.897 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:14:36.897 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:14:36.898 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.898 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.898 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:14:36.898 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:36.898 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.898 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.898 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:14:36.898 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:14:36.898 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.898 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.898 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:14:36.898 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:36.898 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.898 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.899 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:14:36.899 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:14:36.899 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.899 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.899 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:14:36.899 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:36.899 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.899 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.899 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:14:36.899 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:14:36.899 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.899 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.899 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:14:36.899 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:36.899 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.900 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.900 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:14:36.900 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:14:36.900 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.900 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.900 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:14:36.900 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:36.900 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.900 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.900 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:14:36.900 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:14:36.900 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.900 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.900 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:14:36.900 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:36.900 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.900 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.901 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:14:36.901 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:14:36.901 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.901 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.901 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:14:36.901 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:36.901 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.901 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.901 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:14:36.901 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:14:36.901 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.901 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.901 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:14:36.901 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:36.901 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.901 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.902 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:14:36.902 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:14:36.902 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.902 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.902 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:14:36.902 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:36.902 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.902 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.902 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:14:36.902 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:14:36.902 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.902 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.902 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:14:36.902 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:36.902 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.903 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.903 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:14:36.903 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:14:36.903 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.903 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.903 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:14:36.903 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:36.903 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.903 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.903 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:14:36.903 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:14:36.903 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.903 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.903 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:14:36.903 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:36.903 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.904 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.904 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:14:36.904 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:14:36.904 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.904 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:14:36.904 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:14:36.904 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.904 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-22T03:14:36.904 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "peering_event", 2024-09-22T03:14:36.904 INFO:teuthology.orchestra.run.smithi153.stdout: "id": 16777301, 2024-09-22T03:14:36.904 INFO:teuthology.orchestra.run.smithi153.stdout: "detail": { 2024-09-22T03:14:36.904 INFO:teuthology.orchestra.run.smithi153.stdout: "PeeringEvent": { 2024-09-22T03:14:36.904 INFO:teuthology.orchestra.run.smithi153.stdout: "from": "0", 2024-09-22T03:14:36.904 INFO:teuthology.orchestra.run.smithi153.stdout: "pgid": "5.a", 2024-09-22T03:14:36.904 INFO:teuthology.orchestra.run.smithi153.stdout: "sent": 73, 2024-09-22T03:14:36.904 INFO:teuthology.orchestra.run.smithi153.stdout: "requested": 73, 2024-09-22T03:14:36.904 INFO:teuthology.orchestra.run.smithi153.stdout: "evt": "epoch_sent: 73 epoch_requested: 73 DeleteSome", 2024-09-22T03:14:36.905 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-09-22T03:14:36.905 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-22T03:14:36.905 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:14:36.905 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.905 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.905 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-22T03:14:36.905 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGPeeringPipeline::AwaitMap", 2024-09-22T03:14:36.905 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:14:36.724995+0000" 2024-09-22T03:14:36.905 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.905 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-22T03:14:36.905 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:36.905 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.905 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.905 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-22T03:14:36.905 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:14:36.905 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.906 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.906 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-22T03:14:36.906 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:36.906 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.906 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.906 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-22T03:14:36.906 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGPeeringPipeline::Process", 2024-09-22T03:14:36.906 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:14:36.725050+0000" 2024-09-22T03:14:36.906 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.906 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-22T03:14:36.906 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:36.906 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.906 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.906 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-22T03:14:36.906 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:14:36.907 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.907 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:14:36.907 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-22T03:14:36.907 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:14:36.907 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:14:36.907 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.907 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-22T03:14:36.907 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "peering_event", 2024-09-22T03:14:36.907 INFO:teuthology.orchestra.run.smithi153.stdout: "id": 16777304, 2024-09-22T03:14:36.907 INFO:teuthology.orchestra.run.smithi153.stdout: "detail": { 2024-09-22T03:14:36.907 INFO:teuthology.orchestra.run.smithi153.stdout: "PeeringEvent": { 2024-09-22T03:14:36.907 INFO:teuthology.orchestra.run.smithi153.stdout: "from": "0", 2024-09-22T03:14:36.907 INFO:teuthology.orchestra.run.smithi153.stdout: "pgid": "5.0", 2024-09-22T03:14:36.907 INFO:teuthology.orchestra.run.smithi153.stdout: "sent": 73, 2024-09-22T03:14:36.907 INFO:teuthology.orchestra.run.smithi153.stdout: "requested": 73, 2024-09-22T03:14:36.907 INFO:teuthology.orchestra.run.smithi153.stdout: "evt": "epoch_sent: 73 epoch_requested: 73 DeleteSome", 2024-09-22T03:14:36.908 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-09-22T03:14:36.908 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-22T03:14:36.908 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:14:36.908 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.908 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.908 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-22T03:14:36.908 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGPeeringPipeline::AwaitMap", 2024-09-22T03:14:36.908 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:14:36.736999+0000" 2024-09-22T03:14:36.908 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.908 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-22T03:14:36.908 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:36.908 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.908 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.908 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-22T03:14:36.908 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:14:36.909 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.909 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.909 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-22T03:14:36.909 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:36.909 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.909 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.909 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-22T03:14:36.909 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGPeeringPipeline::Process", 2024-09-22T03:14:36.909 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:14:36.737053+0000" 2024-09-22T03:14:36.909 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.909 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-22T03:14:36.909 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:36.909 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.909 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.909 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-22T03:14:36.909 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:14:36.910 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.910 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:14:36.910 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-22T03:14:36.910 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:14:36.910 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:14:36.910 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.910 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-22T03:14:36.910 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "peering_event", 2024-09-22T03:14:36.910 INFO:teuthology.orchestra.run.smithi153.stdout: "id": 33554481, 2024-09-22T03:14:36.910 INFO:teuthology.orchestra.run.smithi153.stdout: "detail": { 2024-09-22T03:14:36.910 INFO:teuthology.orchestra.run.smithi153.stdout: "PeeringEvent": { 2024-09-22T03:14:36.910 INFO:teuthology.orchestra.run.smithi153.stdout: "from": "0", 2024-09-22T03:14:36.911 INFO:teuthology.orchestra.run.smithi153.stdout: "pgid": "5.4", 2024-09-22T03:14:36.911 INFO:teuthology.orchestra.run.smithi153.stdout: "sent": 73, 2024-09-22T03:14:36.911 INFO:teuthology.orchestra.run.smithi153.stdout: "requested": 73, 2024-09-22T03:14:36.911 INFO:teuthology.orchestra.run.smithi153.stdout: "evt": "epoch_sent: 73 epoch_requested: 73 DeleteSome", 2024-09-22T03:14:36.911 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-09-22T03:14:36.911 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-22T03:14:36.911 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:14:36.911 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.911 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.911 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-22T03:14:36.911 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGPeeringPipeline::AwaitMap", 2024-09-22T03:14:36.911 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:14:36.791286+0000" 2024-09-22T03:14:36.911 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.911 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-22T03:14:36.911 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:36.912 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.912 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.912 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-22T03:14:36.912 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:14:36.912 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.912 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.912 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-22T03:14:36.912 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:36.912 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.912 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.912 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-22T03:14:36.912 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGPeeringPipeline::Process", 2024-09-22T03:14:36.912 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:14:36.791327+0000" 2024-09-22T03:14:36.912 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.912 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-22T03:14:36.912 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:14:36.913 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.913 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:14:36.913 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-22T03:14:36.913 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:14:36.913 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:14:36.913 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:14:36.913 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-22T03:14:36.913 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:14:36.913 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:14:36.913 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:14:36.913 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-22T03:14:36.913 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-22T03:14:36.914 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:14:36.860+0000 7f3e09997640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:14:36.914 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:14:36.865+0000 7f3e09997640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:14:37.013 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 57 16 4908 4892 0.670429 0.609375 0.0115624 0.183899 2024-09-22T03:14:38.014 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 58 16 4908 4892 0.65887 0 - 0.183899 2024-09-22T03:14:38.972 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-22T03:14:39.014 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 59 16 5028 5012 0.663591 0.46875 0.0420013 0.188293 2024-09-22T03:14:39.112 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:14:39.108+0000 7f4b9906c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:14:39.118 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:14:39.116+0000 7f4b9906c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:14:40.014 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:2024-09-22T03:14:40.013441+0000 min lat: 0.00692909 max lat: 42.0561 avg lat: 0.181476 2024-09-22T03:14:40.014 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-22T03:14:40.014 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 60 16 5304 5288 0.688465 2.15625 0.0407196 0.181476 2024-09-22T03:14:41.014 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 61 16 5348 5332 0.682813 0.34375 1.02648 0.18085 2024-09-22T03:14:41.335 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-22T03:14:41.440 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:14:41.437+0000 7f89bbe2c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:14:41.443 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:14:41.442+0000 7f89bbe2c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:14:41.914 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [1, 3, 2, 0] 2024-09-22T03:14:41.915 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-22T03:14:41.915 INFO:tasks.thrashosds.thrasher:Killing osd 3, live_osds are [1, 3, 2, 0] 2024-09-22T03:14:41.915 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-09-22T03:14:41.915 INFO:teuthology.orchestra.run:waiting for 300 2024-09-22T03:14:42.014 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 62 16 5404 5388 0.678856 0.4375 0.032761 0.182794 2024-09-22T03:14:43.014 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 63 16 5416 5400 0.669568 0.09375 0.0208857 0.18247 2024-09-22T03:14:43.508 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-22T03:14:43.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:14:43.612+0000 7f41c1cd7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:14:43.618 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:14:43.617+0000 7f41c1cd7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:14:44.014 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 64 16 5416 5400 0.659106 0 - 0.18247 2024-09-22T03:14:45.014 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 65 16 5416 5400 0.648966 0 - 0.18247 2024-09-22T03:14:45.287 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~0s 2024-09-22T03:14:46.015 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 66 16 5416 5400 0.639133 0 - 0.18247 2024-09-22T03:14:47.015 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 67 16 5416 5400 0.629593 0 - 0.18247 2024-09-22T03:14:48.015 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 68 16 5416 5400 0.620334 0 - 0.18247 2024-09-22T03:14:48.016 INFO:tasks.ceph.osd.3:Stopped 2024-09-22T03:14:49.015 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 69 16 5416 5400 0.611344 0 - 0.18247 2024-09-22T03:14:50.015 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 70 16 5416 5400 0.60261 0 - 0.18247 2024-09-22T03:14:50.791 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 has been restored 2024-09-22T03:14:51.015 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 71 16 5416 5400 0.594122 0 - 0.18247 2024-09-22T03:14:52.015 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 72 16 5416 5400 0.58587 0 - 0.18247 2024-09-22T03:14:53.015 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 73 16 5416 5400 0.577844 0 - 0.18247 2024-09-22T03:14:53.017 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [3] live_osds: [1, 2, 0] 2024-09-22T03:14:53.017 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-22T03:14:53.017 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-09-22T03:14:53.017 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-09-22T03:14:53.136 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:14:53.129+0000 7fb7592fa640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:14:53.139 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:14:53.137+0000 7fb7592fa640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:14:53.151 INFO:teuthology.orchestra.run.smithi153.stderr:no valid command found; 10 closest matches: 2024-09-22T03:14:53.151 INFO:teuthology.orchestra.run.smithi153.stderr:config get 2024-09-22T03:14:53.151 INFO:teuthology.orchestra.run.smithi153.stderr:config help 2024-09-22T03:14:53.151 INFO:teuthology.orchestra.run.smithi153.stderr:config set ... 2024-09-22T03:14:53.151 INFO:teuthology.orchestra.run.smithi153.stderr:config show 2024-09-22T03:14:53.151 INFO:teuthology.orchestra.run.smithi153.stderr:deep_scrub 2024-09-22T03:14:53.151 INFO:teuthology.orchestra.run.smithi153.stderr:dump_historic_ops 2024-09-22T03:14:53.152 INFO:teuthology.orchestra.run.smithi153.stderr:dump_historic_slow_ops 2024-09-22T03:14:53.152 INFO:teuthology.orchestra.run.smithi153.stderr:dump_metrics [] 2024-09-22T03:14:53.152 INFO:teuthology.orchestra.run.smithi153.stderr:dump_ops_in_flight 2024-09-22T03:14:53.152 INFO:teuthology.orchestra.run.smithi153.stderr:dump_pgstate_history 2024-09-22T03:14:53.152 INFO:teuthology.orchestra.run.smithi153.stderr:Error EINVAL: invalid command 2024-09-22T03:14:53.154 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-22T03:14:53.154 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-22T03:14:53.155 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-09-22T03:14:53.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:14:53.253+0000 7f0ca0f57640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:14:53.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:14:53.261+0000 7f0ca0f57640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:14:53.277 INFO:teuthology.orchestra.run.smithi153.stderr:no valid command found; 10 closest matches: 2024-09-22T03:14:53.277 INFO:teuthology.orchestra.run.smithi153.stderr:config get 2024-09-22T03:14:53.277 INFO:teuthology.orchestra.run.smithi153.stderr:config help 2024-09-22T03:14:53.277 INFO:teuthology.orchestra.run.smithi153.stderr:config set ... 2024-09-22T03:14:53.277 INFO:teuthology.orchestra.run.smithi153.stderr:config show 2024-09-22T03:14:53.277 INFO:teuthology.orchestra.run.smithi153.stderr:deep_scrub 2024-09-22T03:14:53.277 INFO:teuthology.orchestra.run.smithi153.stderr:dump_historic_ops 2024-09-22T03:14:53.277 INFO:teuthology.orchestra.run.smithi153.stderr:dump_historic_slow_ops 2024-09-22T03:14:53.277 INFO:teuthology.orchestra.run.smithi153.stderr:dump_metrics [] 2024-09-22T03:14:53.277 INFO:teuthology.orchestra.run.smithi153.stderr:dump_ops_in_flight 2024-09-22T03:14:53.277 INFO:teuthology.orchestra.run.smithi153.stderr:dump_pgstate_history 2024-09-22T03:14:53.278 INFO:teuthology.orchestra.run.smithi153.stderr:Error EINVAL: invalid command 2024-09-22T03:14:53.282 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-22T03:14:53.282 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-22T03:14:53.282 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-09-22T03:14:53.387 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:14:53.381+0000 7f70c1157640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:14:53.390 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:14:53.389+0000 7f70c1157640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:14:53.403 INFO:teuthology.orchestra.run.smithi153.stderr:no valid command found; 10 closest matches: 2024-09-22T03:14:53.403 INFO:teuthology.orchestra.run.smithi153.stderr:config get 2024-09-22T03:14:53.403 INFO:teuthology.orchestra.run.smithi153.stderr:config help 2024-09-22T03:14:53.403 INFO:teuthology.orchestra.run.smithi153.stderr:config set ... 2024-09-22T03:14:53.403 INFO:teuthology.orchestra.run.smithi153.stderr:config show 2024-09-22T03:14:53.404 INFO:teuthology.orchestra.run.smithi153.stderr:deep_scrub 2024-09-22T03:14:53.404 INFO:teuthology.orchestra.run.smithi153.stderr:dump_historic_ops 2024-09-22T03:14:53.404 INFO:teuthology.orchestra.run.smithi153.stderr:dump_historic_slow_ops 2024-09-22T03:14:53.404 INFO:teuthology.orchestra.run.smithi153.stderr:dump_metrics [] 2024-09-22T03:14:53.404 INFO:teuthology.orchestra.run.smithi153.stderr:dump_ops_in_flight 2024-09-22T03:14:53.404 INFO:teuthology.orchestra.run.smithi153.stderr:dump_pgstate_history 2024-09-22T03:14:53.404 INFO:teuthology.orchestra.run.smithi153.stderr:Error EINVAL: invalid command 2024-09-22T03:14:53.406 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-22T03:14:53.406 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-22T03:14:53.407 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-09-22T03:14:53.511 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:14:53.505+0000 7feb64397640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:14:53.514 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:14:53.513+0000 7feb64397640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:14:54.016 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 74 16 5416 5400 0.570036 0 - 0.18247 2024-09-22T03:14:55.016 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 75 16 5416 5400 0.562435 0 - 0.18247 2024-09-22T03:14:56.016 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 76 16 5416 5400 0.555034 0 - 0.18247 2024-09-22T03:14:57.016 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 77 16 5416 5400 0.547826 0 - 0.18247 2024-09-22T03:14:58.016 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 78 16 5416 5400 0.540803 0 - 0.18247 2024-09-22T03:14:59.016 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 79 16 5416 5400 0.533957 0 - 0.18247 2024-09-22T03:15:00.017 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:2024-09-22T03:15:00.016056+0000 min lat: 0.00692909 max lat: 42.0561 avg lat: 0.18247 2024-09-22T03:15:00.017 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-22T03:15:00.017 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 80 16 5416 5400 0.527282 0 - 0.18247 2024-09-22T03:15:01.017 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 81 16 5416 5400 0.520772 0 - 0.18247 2024-09-22T03:15:02.017 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 82 16 5416 5400 0.514421 0 - 0.18247 2024-09-22T03:15:03.017 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 83 16 5416 5400 0.508223 0 - 0.18247 2024-09-22T03:15:04.017 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 84 16 5416 5400 0.502173 0 - 0.18247 2024-09-22T03:15:05.017 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 85 16 5416 5400 0.496265 0 - 0.18247 2024-09-22T03:15:06.017 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 86 16 5416 5400 0.490494 0 - 0.18247 2024-09-22T03:15:07.018 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 87 16 5416 5400 0.484856 0 - 0.18247 2024-09-22T03:15:08.017 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 88 16 5416 5400 0.479346 0 - 0.18247 2024-09-22T03:15:09.018 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 89 16 5416 5400 0.47396 0 - 0.18247 2024-09-22T03:15:09.934 INFO:teuthology.orchestra.run.smithi153.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-09-22T03:15:09.934 INFO:teuthology.orchestra.run.smithi153.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-09-22T03:15:09.939 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-22T03:15:09.940 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-22T03:15:09.940 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-22T03:15:09.941 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-22T03:15:10.018 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 90 16 5416 5400 0.468694 0 - 0.18247 2024-09-22T03:15:11.018 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 91 15 5416 5401 0.46363 0.000279018 29.8725 0.187968 2024-09-22T03:15:11.941 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-22T03:15:12.018 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:Total time run: 91.0562 2024-09-22T03:15:12.018 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:Total writes made: 5416 2024-09-22T03:15:12.018 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:Write size: 8192 2024-09-22T03:15:12.018 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:Object size: 65536 2024-09-22T03:15:12.018 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:Bandwidth (MB/sec): 0.464685 2024-09-22T03:15:12.019 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:Stddev Bandwidth: 0.95387 2024-09-22T03:15:12.019 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:Max bandwidth (MB/sec): 2.95312 2024-09-22T03:15:12.019 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:Min bandwidth (MB/sec): 0 2024-09-22T03:15:12.019 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:Average IOPS: 59 2024-09-22T03:15:12.019 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:Stddev IOPS: 122.098 2024-09-22T03:15:12.019 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:Max IOPS: 378 2024-09-22T03:15:12.019 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:Min IOPS: 0 2024-09-22T03:15:12.019 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:Average Latency(s): 0.268956 2024-09-22T03:15:12.019 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:Stddev Latency(s): 2.52771 2024-09-22T03:15:12.019 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:Max latency(s): 42.0561 2024-09-22T03:15:12.019 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:Min latency(s): 0.00692909 2024-09-22T03:15:12.032 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:Cleaning up (deleting benchmark objects) 2024-09-22T03:15:12.056 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:15:12.053+0000 7ff5c0997640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:15:12.063 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:15:12.062+0000 7ff5c0997640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:15:12.262 INFO:teuthology.orchestra.run.smithi153.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-09-22T03:15:12.266 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-22T03:15:12.267 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-22T03:15:13.716 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:Removed 677 objects 2024-09-22T03:15:13.716 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:Clean up completed and total clean up time :1.68332 2024-09-22T03:15:14.268 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-22T03:15:14.372 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:15:14.369+0000 7f9d1c197640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:15:14.375 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:15:14.373+0000 7f9d1c197640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:15:14.418 INFO:teuthology.orchestra.run.smithi153.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-09-22T03:15:14.422 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-22T03:15:14.422 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-22T03:15:14.941 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [3] live_osds: [1, 2, 0] 2024-09-22T03:15:14.941 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-22T03:15:14.941 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-09-22T03:15:14.942 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-09-22T03:15:15.047 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:15:15.041+0000 7fb914e2c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:15:15.050 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:15:15.049+0000 7fb914e2c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:15:15.063 INFO:teuthology.orchestra.run.smithi153.stderr:no valid command found; 10 closest matches: 2024-09-22T03:15:15.063 INFO:teuthology.orchestra.run.smithi153.stderr:config get 2024-09-22T03:15:15.063 INFO:teuthology.orchestra.run.smithi153.stderr:config help 2024-09-22T03:15:15.063 INFO:teuthology.orchestra.run.smithi153.stderr:config set ... 2024-09-22T03:15:15.063 INFO:teuthology.orchestra.run.smithi153.stderr:config show 2024-09-22T03:15:15.063 INFO:teuthology.orchestra.run.smithi153.stderr:deep_scrub 2024-09-22T03:15:15.063 INFO:teuthology.orchestra.run.smithi153.stderr:dump_historic_ops 2024-09-22T03:15:15.063 INFO:teuthology.orchestra.run.smithi153.stderr:dump_historic_slow_ops 2024-09-22T03:15:15.063 INFO:teuthology.orchestra.run.smithi153.stderr:dump_metrics [] 2024-09-22T03:15:15.064 INFO:teuthology.orchestra.run.smithi153.stderr:dump_ops_in_flight 2024-09-22T03:15:15.064 INFO:teuthology.orchestra.run.smithi153.stderr:dump_pgstate_history 2024-09-22T03:15:15.064 INFO:teuthology.orchestra.run.smithi153.stderr:Error EINVAL: invalid command 2024-09-22T03:15:15.066 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-22T03:15:15.067 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-22T03:15:15.067 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-09-22T03:15:15.171 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:15:15.165+0000 7f7a7e157640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:15:15.174 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:15:15.173+0000 7f7a7e157640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:15:15.186 INFO:teuthology.orchestra.run.smithi153.stderr:no valid command found; 10 closest matches: 2024-09-22T03:15:15.187 INFO:teuthology.orchestra.run.smithi153.stderr:config get 2024-09-22T03:15:15.187 INFO:teuthology.orchestra.run.smithi153.stderr:config help 2024-09-22T03:15:15.187 INFO:teuthology.orchestra.run.smithi153.stderr:config set ... 2024-09-22T03:15:15.187 INFO:teuthology.orchestra.run.smithi153.stderr:config show 2024-09-22T03:15:15.187 INFO:teuthology.orchestra.run.smithi153.stderr:deep_scrub 2024-09-22T03:15:15.187 INFO:teuthology.orchestra.run.smithi153.stderr:dump_historic_ops 2024-09-22T03:15:15.187 INFO:teuthology.orchestra.run.smithi153.stderr:dump_historic_slow_ops 2024-09-22T03:15:15.187 INFO:teuthology.orchestra.run.smithi153.stderr:dump_metrics [] 2024-09-22T03:15:15.187 INFO:teuthology.orchestra.run.smithi153.stderr:dump_ops_in_flight 2024-09-22T03:15:15.187 INFO:teuthology.orchestra.run.smithi153.stderr:dump_pgstate_history 2024-09-22T03:15:15.187 INFO:teuthology.orchestra.run.smithi153.stderr:Error EINVAL: invalid command 2024-09-22T03:15:15.190 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-22T03:15:15.190 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-22T03:15:15.191 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-09-22T03:15:15.296 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:15:15.290+0000 7feb96423640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:15:15.299 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:15:15.297+0000 7feb96423640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:15:15.312 INFO:teuthology.orchestra.run.smithi153.stderr:no valid command found; 10 closest matches: 2024-09-22T03:15:15.312 INFO:teuthology.orchestra.run.smithi153.stderr:config get 2024-09-22T03:15:15.312 INFO:teuthology.orchestra.run.smithi153.stderr:config help 2024-09-22T03:15:15.312 INFO:teuthology.orchestra.run.smithi153.stderr:config set ... 2024-09-22T03:15:15.312 INFO:teuthology.orchestra.run.smithi153.stderr:config show 2024-09-22T03:15:15.312 INFO:teuthology.orchestra.run.smithi153.stderr:deep_scrub 2024-09-22T03:15:15.312 INFO:teuthology.orchestra.run.smithi153.stderr:dump_historic_ops 2024-09-22T03:15:15.313 INFO:teuthology.orchestra.run.smithi153.stderr:dump_historic_slow_ops 2024-09-22T03:15:15.313 INFO:teuthology.orchestra.run.smithi153.stderr:dump_metrics [] 2024-09-22T03:15:15.313 INFO:teuthology.orchestra.run.smithi153.stderr:dump_ops_in_flight 2024-09-22T03:15:15.313 INFO:teuthology.orchestra.run.smithi153.stderr:dump_pgstate_history 2024-09-22T03:15:15.313 INFO:teuthology.orchestra.run.smithi153.stderr:Error EINVAL: invalid command 2024-09-22T03:15:15.315 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-22T03:15:15.316 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-22T03:15:15.317 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-09-22T03:15:15.421 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:15:15.415+0000 7fb5dfb97640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:15:15.424 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:15:15.423+0000 7fb5dfb97640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:15:15.428 INFO:teuthology.orchestra.run.smithi153.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-09-22T03:15:15.431 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-22T03:15:15.431 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-22T03:15:16.423 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-22T03:15:16.540 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:15:16.537+0000 7f13eb62c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:15:16.544 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:15:16.542+0000 7f13eb62c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:15:16.589 INFO:teuthology.orchestra.run.smithi153.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-09-22T03:15:16.593 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-22T03:15:16.593 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-22T03:15:17.570 INFO:tasks.ceph.ceph_manager.ceph:removing pool_name unique_pool_3 2024-09-22T03:15:17.570 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm unique_pool_3 unique_pool_3 --yes-i-really-really-mean-it 2024-09-22T03:15:17.694 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:15:17.684+0000 7f2661e6c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:15:17.697 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:15:17.695+0000 7f2661e6c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:15:17.948 INFO:teuthology.orchestra.run.smithi153.stderr:pool 'unique_pool_3' removed 2024-09-22T03:15:17.968 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-09-22T03:15:17.968 INFO:tasks.radosbench:Beginning radosbench... 2024-09-22T03:15:17.968 INFO:tasks.ceph.ceph_manager.ceph:creating pool_name unique_pool_4 2024-09-22T03:15:17.969 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create unique_pool_4 16 2024-09-22T03:15:18.115 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:15:18.106+0000 7fa7ea86c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:15:18.120 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:15:18.118+0000 7fa7ea86c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:15:18.594 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-22T03:15:18.699 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:15:18.696+0000 7f03bd0b2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:15:18.702 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:15:18.701+0000 7f03bd0b2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:15:18.748 INFO:teuthology.orchestra.run.smithi153.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-09-22T03:15:18.751 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-22T03:15:18.751 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-22T03:15:18.899 INFO:teuthology.orchestra.run.smithi153.stderr:pool 'unique_pool_4' created 2024-09-22T03:15:18.914 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool application enable unique_pool_4 rados --yes-i-really-mean-it || true 2024-09-22T03:15:19.041 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:15:19.031+0000 7f6ee14d7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:15:19.044 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:15:19.043+0000 7f6ee14d7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:15:19.900 INFO:teuthology.orchestra.run.smithi153.stderr:enabled application 'rados' on pool 'unique_pool_4' 2024-09-22T03:15:20.432 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [3] live_osds: [1, 2, 0] 2024-09-22T03:15:20.432 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-22T03:15:20.433 INFO:tasks.thrashosds.thrasher:Reviving osd 3 2024-09-22T03:15:20.433 INFO:tasks.ceph.osd.3:Restarting daemon 2024-09-22T03:15:20.433 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2>> /var/log/ceph/ceph-osd.3.log 2024-09-22T03:15:20.436 INFO:tasks.ceph.osd.3:Started 2024-09-22T03:15:20.436 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-22T03:15:20.548 INFO:teuthology.orchestra.run.smithi196.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-09-22T03:15:20.551 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-22T03:15:20.551 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-09-22T03:15:20.753 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-22T03:15:20.860 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:15:20.857+0000 7f8528d17640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:15:20.863 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:15:20.862+0000 7f8528d17640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:15:20.908 INFO:teuthology.orchestra.run.smithi153.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-09-22T03:15:20.914 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-22T03:15:20.914 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-22T03:15:20.922 DEBUG:teuthology.orchestra.run.smithi153:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 8192 --object-size 65536 -p unique_pool_4 bench 90 write' 2024-09-22T03:15:20.924 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-09-22T03:15:20.924 INFO:teuthology.orchestra.run:waiting for 3000 2024-09-22T03:15:20.979 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:hints = 1 2024-09-22T03:15:20.980 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:Maintaining 16 concurrent writes of 8192 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-09-22T03:15:20.980 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:Object prefix: benchmark_data_smithi153_37783 2024-09-22T03:15:20.980 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-22T03:15:20.980 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 0 0 0 0 0 0 - 0 2024-09-22T03:15:21.979 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 1 16 414 398 3.1092 3.10938 0.0525093 0.0390268 2024-09-22T03:15:22.915 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-22T03:15:22.980 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 2 16 813 797 3.11307 3.11719 0.023922 0.0396388 2024-09-22T03:15:23.049 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:15:23.046+0000 7f8ab626c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:15:23.054 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:15:23.052+0000 7f8ab626c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:15:23.139 INFO:teuthology.orchestra.run.smithi153.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-09-22T03:15:23.142 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-22T03:15:23.143 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-22T03:15:23.980 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 3 16 1164 1148 2.98935 2.74219 0.055889 0.0414178 2024-09-22T03:15:24.980 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 4 16 1524 1508 2.94508 2.8125 0.0247145 0.0422199 2024-09-22T03:15:25.144 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-22T03:15:25.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:15:25.271+0000 7f8a830d7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:15:25.280 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:15:25.278+0000 7f8a830d7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:15:25.369 INFO:teuthology.orchestra.run.smithi153.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-09-22T03:15:25.373 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-22T03:15:25.374 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-22T03:15:25.552 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-22T03:15:25.661 INFO:teuthology.orchestra.run.smithi196.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-09-22T03:15:25.663 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-22T03:15:25.664 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-09-22T03:15:25.980 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 5 16 1871 1855 2.89821 2.71094 0.0159632 0.0429578 2024-09-22T03:15:26.980 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 6 16 2256 2240 2.91644 3.00781 0.0754007 0.0425803 2024-09-22T03:15:27.374 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-22T03:15:27.520 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:15:27.517+0000 7f0654757640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:15:27.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:15:27.522+0000 7f0654757640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:15:27.646 INFO:teuthology.orchestra.run.smithi153.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-09-22T03:15:27.650 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-22T03:15:27.650 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-22T03:15:27.980 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 7 16 2611 2595 2.89597 2.77344 0.0233702 0.043041 2024-09-22T03:15:28.980 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 8 16 2937 2921 2.8523 2.54688 0.0537264 0.0432881 2024-09-22T03:15:29.651 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-22T03:15:29.756 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:15:29.753+0000 7efeb7d17640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:15:29.759 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:15:29.757+0000 7efeb7d17640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:15:29.980 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 9 16 2937 2921 2.53537 0 - 0.0432881 2024-09-22T03:15:30.664 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-22T03:15:30.767 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-09-22T03:15:30.767 INFO:teuthology.orchestra.run.smithi196.stdout: "ops_in_flight": [] 2024-09-22T03:15:30.767 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-09-22T03:15:30.980 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 10 16 2937 2921 2.28184 0 - 0.0432881 2024-09-22T03:15:31.825 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-22T03:15:31.980 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 11 16 3174 3158 2.24271 0.617188 0.0768761 0.0554942 2024-09-22T03:15:31.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:15:31.981+0000 7f6f6ead7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:15:31.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:15:31.989+0000 7f6f6ead7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:15:32.980 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 12 16 3232 3216 2.09358 0.453125 0.0244712 0.0553482 2024-09-22T03:15:33.980 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 13 16 3248 3232 1.94215 0.125 0.895167 0.0597312 2024-09-22T03:15:34.113 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-22T03:15:34.238 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:15:34.235+0000 7f2a8f797640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:15:34.241 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:15:34.240+0000 7f2a8f797640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:15:34.981 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 14 16 3519 3503 1.95464 2.11719 0.740908 0.0632938 2024-09-22T03:15:35.776 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [1, 2, 0, 3] 2024-09-22T03:15:35.776 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-22T03:15:35.777 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-22T03:15:35.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:15:35.921+0000 7f97abc2c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:15:35.938 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:15:35.937+0000 7f97abc2c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:15:35.981 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 15 16 3901 3885 2.02327 2.98438 0.0437634 0.06169 2024-09-22T03:15:36.221 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-22T03:15:36.221 INFO:teuthology.orchestra.run.smithi153.stderr:dumped all 2024-09-22T03:15:36.238 INFO:teuthology.orchestra.run.smithi153.stdout:{"pg_ready":true,"pg_map":{"version":313,"stamp":"2024-09-22T03:15:36.204882+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":29016627,"num_objects":439,"num_object_clones":0,"num_object_copies":1317,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":14,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":3609,"num_write_kb":29656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":277,"num_bytes_recovered":18071552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3560,"ondisk_log_size":3560,"up":75,"acting":74,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":183532,"kb_used_data":33132,"kb_used_omap":0,"kb_used_meta":150336,"kb_avail":377303828,"statfs":{"total":386547056640,"available":386359119872,"internally_reserved":0,"allocated":33927168,"data_stored":83982983,"data_compressed":4181905,"data_compressed_allocated":30773248,"data_compressed_original":82706432,"omap_allocated":0,"internal_metadata":153944064},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":4849664,"num_objects":75,"num_object_clones":0,"num_object_copies":225,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":592,"num_write_kb":4736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":264,"num_bytes_recovered":17219584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.288001"},"pg_stats":[{"pgid":"7.f","version":"92'191","reported_seq":231,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-22T03:15:34.021226+0000","last_change":"2024-09-22T03:15:33.972925+0000","last_active":"2024-09-22T03:15:34.021226+0000","last_peered":"2024-09-22T03:15:34.021226+0000","last_clean":"2024-09-22T03:15:34.021226+0000","last_became_active":"2024-09-22T03:15:33.967199+0000","last_became_peered":"2024-09-22T03:15:33.967199+0000","last_unstale":"2024-09-22T03:15:34.021226+0000","last_undegraded":"2024-09-22T03:15:34.021226+0000","last_fullsized":"2024-09-22T03:15:34.021226+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_clean_scrub_stamp":"2024-09-22T03:15:18.897706+0000","objects_scrubbed":0,"log_size":191,"log_dups_size":0,"ondisk_log_size":191,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1572864,"num_objects":24,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":192,"num_write_kb":1536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":1245184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.e","version":"90'216","reported_seq":246,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-22T03:15:33.914333+0000","last_change":"2024-09-22T03:15:20.007094+0000","last_active":"2024-09-22T03:15:33.914333+0000","last_peered":"2024-09-22T03:15:33.914333+0000","last_clean":"2024-09-22T03:15:33.914333+0000","last_became_active":"2024-09-22T03:15:20.000757+0000","last_became_peered":"2024-09-22T03:15:20.000757+0000","last_unstale":"2024-09-22T03:15:33.914333+0000","last_undegraded":"2024-09-22T03:15:33.914333+0000","last_fullsized":"2024-09-22T03:15:33.914333+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_clean_scrub_stamp":"2024-09-22T03:15:18.897706+0000","objects_scrubbed":0,"log_size":216,"log_dups_size":0,"ondisk_log_size":216,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1769472,"num_objects":27,"num_object_clones":0,"num_object_copies":81,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":216,"num_write_kb":1728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.d","version":"93'271","reported_seq":362,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-22T03:15:35.660012+0000","last_change":"2024-09-22T03:15:33.956874+0000","last_active":"2024-09-22T03:15:35.660012+0000","last_peered":"2024-09-22T03:15:35.660012+0000","last_clean":"2024-09-22T03:15:35.660012+0000","last_became_active":"2024-09-22T03:15:33.953772+0000","last_became_peered":"2024-09-22T03:15:33.953772+0000","last_unstale":"2024-09-22T03:15:35.660012+0000","last_undegraded":"2024-09-22T03:15:35.660012+0000","last_fullsized":"2024-09-22T03:15:35.660012+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_clean_scrub_stamp":"2024-09-22T03:15:18.897706+0000","objects_scrubbed":0,"log_size":271,"log_dups_size":0,"ondisk_log_size":271,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2228224,"num_objects":34,"num_object_clones":0,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":272,"num_write_kb":2176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":1835008,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.c","version":"93'199","reported_seq":266,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-22T03:15:35.696026+0000","last_change":"2024-09-22T03:15:33.954730+0000","last_active":"2024-09-22T03:15:35.696026+0000","last_peered":"2024-09-22T03:15:35.696026+0000","last_clean":"2024-09-22T03:15:35.696026+0000","last_became_active":"2024-09-22T03:15:33.951105+0000","last_became_peered":"2024-09-22T03:15:33.951105+0000","last_unstale":"2024-09-22T03:15:35.696026+0000","last_undegraded":"2024-09-22T03:15:35.696026+0000","last_fullsized":"2024-09-22T03:15:35.696026+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_clean_scrub_stamp":"2024-09-22T03:15:18.897706+0000","objects_scrubbed":0,"log_size":199,"log_dups_size":0,"ondisk_log_size":199,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1638400,"num_objects":25,"num_object_clones":0,"num_object_copies":75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":200,"num_write_kb":1600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":1048576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.b","version":"86'248","reported_seq":261,"reported_epoch":92,"state":"peering","last_fresh":"2024-09-22T03:15:33.915523+0000","last_change":"2024-09-22T03:15:33.915523+0000","last_active":"2024-09-22T03:15:29.926993+0000","last_peered":"2024-09-22T03:15:28.838069+0000","last_clean":"2024-09-22T03:15:18.897706+0000","last_became_active":"2024-09-22T03:15:19.977280+0000","last_became_peered":"2024-09-22T03:15:19.977280+0000","last_unstale":"2024-09-22T03:15:33.915523+0000","last_undegraded":"2024-09-22T03:15:33.915523+0000","last_fullsized":"2024-09-22T03:15:33.915523+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_clean_scrub_stamp":"2024-09-22T03:15:18.897706+0000","objects_scrubbed":0,"log_size":248,"log_dups_size":0,"ondisk_log_size":248,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2031616,"num_objects":31,"num_object_clones":0,"num_object_copies":93,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":248,"num_write_kb":1984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"86'184","reported_seq":205,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-22T03:15:33.977007+0000","last_change":"2024-09-22T03:15:33.977496+0000","last_active":"2024-09-22T03:15:33.977007+0000","last_peered":"2024-09-22T03:15:33.977007+0000","last_clean":"2024-09-22T03:15:18.897706+0000","last_became_active":"2024-09-22T03:15:33.972140+0000","last_became_peered":"2024-09-22T03:15:33.972140+0000","last_unstale":"2024-09-22T03:15:33.977007+0000","last_undegraded":"2024-09-22T03:15:33.977007+0000","last_fullsized":"2024-09-22T03:15:33.977007+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_clean_scrub_stamp":"2024-09-22T03:15:18.897706+0000","objects_scrubbed":0,"log_size":184,"log_dups_size":0,"ondisk_log_size":184,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1507328,"num_objects":23,"num_object_clones":0,"num_object_copies":69,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":184,"num_write_kb":1472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":1507328,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.9","version":"93'207","reported_seq":278,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-22T03:15:35.830737+0000","last_change":"2024-09-22T03:15:33.948202+0000","last_active":"2024-09-22T03:15:35.830737+0000","last_peered":"2024-09-22T03:15:35.830737+0000","last_clean":"2024-09-22T03:15:35.830737+0000","last_became_active":"2024-09-22T03:15:33.943275+0000","last_became_peered":"2024-09-22T03:15:33.943275+0000","last_unstale":"2024-09-22T03:15:35.830737+0000","last_undegraded":"2024-09-22T03:15:35.830737+0000","last_fullsized":"2024-09-22T03:15:35.830737+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_clean_scrub_stamp":"2024-09-22T03:15:18.897706+0000","objects_scrubbed":0,"log_size":207,"log_dups_size":0,"ondisk_log_size":207,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1703936,"num_objects":26,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":208,"num_write_kb":1664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":1179648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.8","version":"92'207","reported_seq":285,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-22T03:15:34.149171+0000","last_change":"2024-09-22T03:15:33.946927+0000","last_active":"2024-09-22T03:15:34.149171+0000","last_peered":"2024-09-22T03:15:34.149171+0000","last_clean":"2024-09-22T03:15:34.149171+0000","last_became_active":"2024-09-22T03:15:33.940267+0000","last_became_peered":"2024-09-22T03:15:33.940267+0000","last_unstale":"2024-09-22T03:15:34.149171+0000","last_undegraded":"2024-09-22T03:15:34.149171+0000","last_fullsized":"2024-09-22T03:15:34.149171+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_clean_scrub_stamp":"2024-09-22T03:15:18.897706+0000","objects_scrubbed":0,"log_size":207,"log_dups_size":0,"ondisk_log_size":207,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1703936,"num_objects":26,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":208,"num_write_kb":1664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":1507328,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.6","version":"92'252","reported_seq":303,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-22T03:15:34.108101+0000","last_change":"2024-09-22T03:15:32.999617+0000","last_active":"2024-09-22T03:15:34.108101+0000","last_peered":"2024-09-22T03:15:34.108101+0000","last_clean":"2024-09-22T03:15:34.108101+0000","last_became_active":"2024-09-22T03:15:32.993508+0000","last_became_peered":"2024-09-22T03:15:32.993508+0000","last_unstale":"2024-09-22T03:15:34.108101+0000","last_undegraded":"2024-09-22T03:15:34.108101+0000","last_fullsized":"2024-09-22T03:15:34.108101+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_clean_scrub_stamp":"2024-09-22T03:15:18.897706+0000","objects_scrubbed":0,"log_size":252,"log_dups_size":0,"ondisk_log_size":252,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2072576,"num_objects":32,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":253,"num_write_kb":2024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":491520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":112,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-22T03:15:33.917574+0000","last_change":"2024-09-22T03:15:30.926185+0000","last_active":"2024-09-22T03:15:33.917574+0000","last_peered":"2024-09-22T03:15:33.917574+0000","last_clean":"2024-09-22T03:15:33.917574+0000","last_became_active":"2024-09-22T03:15:29.957045+0000","last_became_peered":"2024-09-22T03:15:29.957045+0000","last_unstale":"2024-09-22T03:15:33.917574+0000","last_undegraded":"2024-09-22T03:15:33.917574+0000","last_fullsized":"2024-09-22T03:15:33.917574+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.7","version":"90'248","reported_seq":335,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-22T03:15:33.948255+0000","last_change":"2024-09-22T03:15:33.949109+0000","last_active":"2024-09-22T03:15:33.948255+0000","last_peered":"2024-09-22T03:15:33.948255+0000","last_clean":"2024-09-22T03:15:18.897706+0000","last_became_active":"2024-09-22T03:15:33.943481+0000","last_became_peered":"2024-09-22T03:15:33.943481+0000","last_unstale":"2024-09-22T03:15:33.948255+0000","last_undegraded":"2024-09-22T03:15:33.948255+0000","last_fullsized":"2024-09-22T03:15:33.948255+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_clean_scrub_stamp":"2024-09-22T03:15:18.897706+0000","objects_scrubbed":0,"log_size":248,"log_dups_size":0,"ondisk_log_size":248,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2031616,"num_objects":31,"num_object_clones":0,"num_object_copies":93,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":248,"num_write_kb":1984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":1835008,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":186,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-22T03:15:30.924282+0000","last_change":"2024-09-22T03:14:38.225116+0000","last_active":"2024-09-22T03:15:30.924282+0000","last_peered":"2024-09-22T03:15:30.924282+0000","last_clean":"2024-09-22T03:15:30.924282+0000","last_became_active":"2024-09-22T03:14:37.266115+0000","last_became_peered":"2024-09-22T03:14:37.266115+0000","last_unstale":"2024-09-22T03:15:30.924282+0000","last_undegraded":"2024-09-22T03:15:30.924282+0000","last_fullsized":"2024-09-22T03:15:30.924282+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.4","version":"93'255","reported_seq":332,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-22T03:15:36.014164+0000","last_change":"2024-09-22T03:15:33.941381+0000","last_active":"2024-09-22T03:15:36.014164+0000","last_peered":"2024-09-22T03:15:36.014164+0000","last_clean":"2024-09-22T03:15:36.014164+0000","last_became_active":"2024-09-22T03:15:33.938030+0000","last_became_peered":"2024-09-22T03:15:33.938030+0000","last_unstale":"2024-09-22T03:15:36.014164+0000","last_undegraded":"2024-09-22T03:15:36.014164+0000","last_fullsized":"2024-09-22T03:15:36.014164+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_clean_scrub_stamp":"2024-09-22T03:15:18.897706+0000","objects_scrubbed":0,"log_size":255,"log_dups_size":0,"ondisk_log_size":255,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2097152,"num_objects":32,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":256,"num_write_kb":2048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1376256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":227,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-22T03:15:34.915478+0000","last_change":"2024-09-22T03:15:30.950741+0000","last_active":"2024-09-22T03:15:34.915478+0000","last_peered":"2024-09-22T03:15:34.915478+0000","last_clean":"2024-09-22T03:15:34.915478+0000","last_became_active":"2024-09-22T03:15:29.979306+0000","last_became_peered":"2024-09-22T03:15:29.979306+0000","last_unstale":"2024-09-22T03:15:34.915478+0000","last_undegraded":"2024-09-22T03:15:34.915478+0000","last_fullsized":"2024-09-22T03:15:34.915478+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.5","version":"89'256","reported_seq":348,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-22T03:15:33.911068+0000","last_change":"2024-09-22T03:15:32.978797+0000","last_active":"2024-09-22T03:15:33.911068+0000","last_peered":"2024-09-22T03:15:33.911068+0000","last_clean":"2024-09-22T03:15:33.911068+0000","last_became_active":"2024-09-22T03:15:32.974824+0000","last_became_peered":"2024-09-22T03:15:32.974824+0000","last_unstale":"2024-09-22T03:15:33.911068+0000","last_undegraded":"2024-09-22T03:15:33.911068+0000","last_fullsized":"2024-09-22T03:15:33.911068+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_clean_scrub_stamp":"2024-09-22T03:15:18.897706+0000","objects_scrubbed":0,"log_size":256,"log_dups_size":0,"ondisk_log_size":256,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2097152,"num_objects":32,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":256,"num_write_kb":2048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":1966080,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":173,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-22T03:15:33.914764+0000","last_change":"2024-09-22T03:15:30.924363+0000","last_active":"2024-09-22T03:15:33.914764+0000","last_peered":"2024-09-22T03:15:33.914764+0000","last_clean":"2024-09-22T03:15:33.914764+0000","last_became_active":"2024-09-22T03:15:29.958549+0000","last_became_peered":"2024-09-22T03:15:29.958549+0000","last_unstale":"2024-09-22T03:15:33.914764+0000","last_undegraded":"2024-09-22T03:15:33.914764+0000","last_fullsized":"2024-09-22T03:15:33.914764+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"12'97","reported_seq":309,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-22T03:15:30.945076+0000","last_change":"2024-09-22T03:14:37.276946+0000","last_active":"2024-09-22T03:15:30.945076+0000","last_peered":"2024-09-22T03:15:30.945076+0000","last_clean":"2024-09-22T03:15:30.945076+0000","last_became_active":"2024-09-22T03:14:37.273858+0000","last_became_peered":"2024-09-22T03:14:37.273858+0000","last_unstale":"2024-09-22T03:15:30.945076+0000","last_undegraded":"2024-09-22T03:15:30.945076+0000","last_fullsized":"2024-09-22T03:15:30.945076+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.2","version":"90'192","reported_seq":237,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-22T03:15:33.952666+0000","last_change":"2024-09-22T03:15:33.953595+0000","last_active":"2024-09-22T03:15:33.952666+0000","last_peered":"2024-09-22T03:15:33.952666+0000","last_clean":"2024-09-22T03:15:18.897706+0000","last_became_active":"2024-09-22T03:15:33.947435+0000","last_became_peered":"2024-09-22T03:15:33.947435+0000","last_unstale":"2024-09-22T03:15:33.952666+0000","last_undegraded":"2024-09-22T03:15:33.952666+0000","last_fullsized":"2024-09-22T03:15:33.952666+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_clean_scrub_stamp":"2024-09-22T03:15:18.897706+0000","objects_scrubbed":0,"log_size":192,"log_dups_size":0,"ondisk_log_size":192,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1572864,"num_objects":24,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":192,"num_write_kb":1536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1376256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":200,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-22T03:15:33.913520+0000","last_change":"2024-09-22T03:14:38.265993+0000","last_active":"2024-09-22T03:15:33.913520+0000","last_peered":"2024-09-22T03:15:33.913520+0000","last_clean":"2024-09-22T03:15:33.913520+0000","last_became_active":"2024-09-22T03:14:37.276817+0000","last_became_peered":"2024-09-22T03:14:37.276817+0000","last_unstale":"2024-09-22T03:15:33.913520+0000","last_undegraded":"2024-09-22T03:15:33.913520+0000","last_fullsized":"2024-09-22T03:15:33.913520+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.3","version":"92'191","reported_seq":233,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-22T03:15:34.089250+0000","last_change":"2024-09-22T03:15:33.961634+0000","last_active":"2024-09-22T03:15:34.089250+0000","last_peered":"2024-09-22T03:15:34.089250+0000","last_clean":"2024-09-22T03:15:34.089250+0000","last_became_active":"2024-09-22T03:15:33.957171+0000","last_became_peered":"2024-09-22T03:15:33.957171+0000","last_unstale":"2024-09-22T03:15:34.089250+0000","last_undegraded":"2024-09-22T03:15:34.089250+0000","last_fullsized":"2024-09-22T03:15:34.089250+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_clean_scrub_stamp":"2024-09-22T03:15:18.897706+0000","objects_scrubbed":0,"log_size":191,"log_dups_size":0,"ondisk_log_size":191,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1572864,"num_objects":24,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":192,"num_write_kb":1536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1376256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":206,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-22T03:15:34.915279+0000","last_change":"2024-09-22T03:15:30.950892+0000","last_active":"2024-09-22T03:15:34.915279+0000","last_peered":"2024-09-22T03:15:34.915279+0000","last_clean":"2024-09-22T03:15:34.915279+0000","last_became_active":"2024-09-22T03:15:29.963452+0000","last_became_peered":"2024-09-22T03:15:29.963452+0000","last_unstale":"2024-09-22T03:15:34.915279+0000","last_undegraded":"2024-09-22T03:15:34.915279+0000","last_fullsized":"2024-09-22T03:15:34.915279+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.0","version":"93'224","reported_seq":304,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-22T03:15:35.470189+0000","last_change":"2024-09-22T03:15:32.976320+0000","last_active":"2024-09-22T03:15:35.470189+0000","last_peered":"2024-09-22T03:15:35.470189+0000","last_clean":"2024-09-22T03:15:35.470189+0000","last_became_active":"2024-09-22T03:15:32.973547+0000","last_became_peered":"2024-09-22T03:15:32.973547+0000","last_unstale":"2024-09-22T03:15:35.470189+0000","last_undegraded":"2024-09-22T03:15:35.470189+0000","last_fullsized":"2024-09-22T03:15:35.470189+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_clean_scrub_stamp":"2024-09-22T03:15:18.897706+0000","objects_scrubbed":0,"log_size":224,"log_dups_size":0,"ondisk_log_size":224,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1843200,"num_objects":29,"num_object_clones":0,"num_object_copies":87,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":225,"num_write_kb":1800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1327104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":214,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-22T03:15:33.914578+0000","last_change":"2024-09-22T03:15:30.958044+0000","last_active":"2024-09-22T03:15:33.914578+0000","last_peered":"2024-09-22T03:15:33.914578+0000","last_clean":"2024-09-22T03:15:33.914578+0000","last_became_active":"2024-09-22T03:15:29.976915+0000","last_became_peered":"2024-09-22T03:15:29.976915+0000","last_unstale":"2024-09-22T03:15:33.914578+0000","last_undegraded":"2024-09-22T03:15:33.914578+0000","last_fullsized":"2024-09-22T03:15:33.914578+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1","version":"89'119","reported_seq":138,"reported_epoch":89,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-22T03:15:31.194608+0000","last_change":"2024-09-22T03:15:31.158862+0000","last_active":"2024-09-22T03:15:31.194608+0000","last_peered":"2024-09-22T03:15:31.194608+0000","last_clean":"2024-09-22T03:15:18.897706+0000","last_became_active":"2024-09-22T03:15:31.151443+0000","last_became_peered":"2024-09-22T03:15:31.151443+0000","last_unstale":"2024-09-22T03:15:31.194608+0000","last_undegraded":"2024-09-22T03:15:30.959502+0000","last_fullsized":"2024-09-22T03:15:30.953960+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_clean_scrub_stamp":"2024-09-22T03:15:18.897706+0000","objects_scrubbed":0,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":983040,"num_objects":15,"num_object_clones":0,"num_object_copies":45,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":14,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":209,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-22T03:15:33.916371+0000","last_change":"2024-09-22T03:15:30.958189+0000","last_active":"2024-09-22T03:15:33.916371+0000","last_peered":"2024-09-22T03:15:33.916371+0000","last_clean":"2024-09-22T03:15:33.916371+0000","last_became_active":"2024-09-22T03:15:29.980027+0000","last_became_peered":"2024-09-22T03:15:29.980027+0000","last_unstale":"2024-09-22T03:15:33.916371+0000","last_undegraded":"2024-09-22T03:15:33.916371+0000","last_fullsized":"2024-09-22T03:15:33.916371+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":28426240,"num_objects":435,"num_object_clones":0,"num_object_copies":1305,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":14,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3470,"num_write_kb":27760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":277,"num_bytes_recovered":18071552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":31485952,"data_stored":82657280,"data_compressed":4178311,"data_compressed_allocated":31485952,"data_compressed_original":82657280,"omap_allocated":0,"internal_metadata":0},"log_size":3460,"ondisk_log_size":3460,"up":48,"acting":47,"num_store_stats":4},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1180736,"data_compressed":59924,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":87,"seq":373662154757,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":40684,"kb_used_data":3228,"kb_used_omap":0,"kb_used_meta":37440,"kb_avail":94331156,"statfs":{"total":96636764160,"available":96595103744,"internally_reserved":0,"allocated":3305472,"data_stored":25015342,"data_compressed":1188467,"data_compressed_allocated":2527232,"data_compressed_original":24707072,"omap_allocated":0,"internal_metadata":38338560},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":71,"seq":304942678091,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":45536,"kb_used_data":9424,"kb_used_omap":0,"kb_used_meta":36096,"kb_avail":94326304,"statfs":{"total":96636764160,"available":96590135296,"internally_reserved":0,"allocated":9650176,"data_stored":18506519,"data_compressed":938351,"data_compressed_allocated":8843264,"data_compressed_original":18178048,"omap_allocated":0,"internal_metadata":36962304},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":53,"seq":227633266867,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":56200,"kb_used_data":11512,"kb_used_omap":0,"kb_used_meta":44672,"kb_avail":94315640,"statfs":{"total":96636764160,"available":96579215360,"internally_reserved":0,"allocated":11788288,"data_stored":22804046,"data_compressed":1160161,"data_compressed_allocated":10989568,"data_compressed_original":22478848,"omap_allocated":0,"internal_metadata":45744128},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":73,"seq":313532612663,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41112,"kb_used_data":8968,"kb_used_omap":0,"kb_used_meta":32128,"kb_avail":94330728,"statfs":{"total":96636764160,"available":96594665472,"internally_reserved":0,"allocated":9183232,"data_stored":17657076,"data_compressed":894926,"data_compressed_allocated":8413184,"data_compressed_original":17342464,"omap_allocated":0,"internal_metadata":32899072},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29962,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29962,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8826880,"data_stored":17678336,"data_compressed":911917,"data_compressed_allocated":8826880,"data_compressed_original":17678336,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":11112448,"data_stored":22233088,"data_compressed":1147119,"data_compressed_allocated":11112448,"data_compressed_original":22233088,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8867840,"data_stored":17735680,"data_compressed":915157,"data_compressed_allocated":8867840,"data_compressed_original":17735680,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2678784,"data_stored":25010176,"data_compressed":1204118,"data_compressed_allocated":2678784,"data_compressed_original":25010176,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-22T03:15:36.348 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-22T03:15:36.474 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:15:36.470+0000 7f4e418fa640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:15:36.479 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:15:36.477+0000 7f4e418fa640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:15:36.981 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 16 16 4228 4212 2.05647 2.55469 0.0422278 0.0606607 2024-09-22T03:15:37.981 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 17 16 4554 4538 2.08531 2.54688 0.0374283 0.0598276 2024-09-22T03:15:38.695 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-22T03:15:38.843 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:15:38.839+0000 7fe796757640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:15:38.847 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:15:38.846+0000 7fe796757640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:15:38.981 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 18 16 4873 4857 2.1079 2.49219 0.0221397 0.0592089 2024-09-22T03:15:39.981 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 19 16 5192 5176 2.12811 2.49219 0.0476319 0.0586261 2024-09-22T03:15:40.981 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:2024-09-22T03:15:40.980738+0000 min lat: 0.00692898 max lat: 2.41342 avg lat: 0.057862 2024-09-22T03:15:40.981 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-22T03:15:40.981 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 20 16 5538 5522 2.15685 2.70312 0.0764419 0.057862 2024-09-22T03:15:40.990 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-22T03:15:41.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:15:41.108+0000 7f8153357640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:15:41.116 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:15:41.114+0000 7f8153357640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:15:41.240 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [1, 2, 0, 3] 2024-09-22T03:15:41.240 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-22T03:15:41.240 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-22T03:15:41.376 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:15:41.370+0000 7fc928463640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:15:41.379 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:15:41.378+0000 7fc928463640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:15:41.663 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-22T03:15:41.663 INFO:teuthology.orchestra.run.smithi153.stderr:dumped all 2024-09-22T03:15:41.681 INFO:teuthology.orchestra.run.smithi153.stdout:{"pg_ready":true,"pg_map":{"version":316,"stamp":"2024-09-22T03:15:40.205648+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":38330931,"num_objects":582,"num_object_clones":0,"num_object_copies":1746,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":4746,"num_write_kb":38752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":322,"num_bytes_recovered":21020672,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4696,"ondisk_log_size":4696,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":193756,"kb_used_data":47516,"kb_used_omap":0,"kb_used_meta":146176,"kb_avail":377293604,"statfs":{"total":386547056640,"available":386348650496,"internally_reserved":0,"allocated":48656384,"data_stored":113337259,"data_compressed":5696245,"data_compressed_allocated":45436928,"data_compressed_original":112033792,"omap_allocated":0,"internal_metadata":149684224},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":14163968,"num_objects":218,"num_object_clones":0,"num_object_copies":654,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1729,"num_write_kb":13832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":309,"num_bytes_recovered":20168704,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.303559"},"pg_stats":[{"pgid":"7.f","version":"93'311","reported_seq":357,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-22T03:15:38.934046+0000","last_change":"2024-09-22T03:15:33.972925+0000","last_active":"2024-09-22T03:15:38.934046+0000","last_peered":"2024-09-22T03:15:38.934046+0000","last_clean":"2024-09-22T03:15:38.934046+0000","last_became_active":"2024-09-22T03:15:33.967199+0000","last_became_peered":"2024-09-22T03:15:33.967199+0000","last_unstale":"2024-09-22T03:15:38.934046+0000","last_undegraded":"2024-09-22T03:15:38.934046+0000","last_fullsized":"2024-09-22T03:15:38.934046+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_clean_scrub_stamp":"2024-09-22T03:15:18.897706+0000","objects_scrubbed":0,"log_size":311,"log_dups_size":0,"ondisk_log_size":311,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2555904,"num_objects":39,"num_object_clones":0,"num_object_copies":117,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":312,"num_write_kb":2496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":1245184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.e","version":"93'296","reported_seq":332,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-22T03:15:38.938768+0000","last_change":"2024-09-22T03:15:20.007094+0000","last_active":"2024-09-22T03:15:38.938768+0000","last_peered":"2024-09-22T03:15:38.938768+0000","last_clean":"2024-09-22T03:15:38.938768+0000","last_became_active":"2024-09-22T03:15:20.000757+0000","last_became_peered":"2024-09-22T03:15:20.000757+0000","last_unstale":"2024-09-22T03:15:38.938768+0000","last_undegraded":"2024-09-22T03:15:38.938768+0000","last_fullsized":"2024-09-22T03:15:38.938768+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_clean_scrub_stamp":"2024-09-22T03:15:18.897706+0000","objects_scrubbed":0,"log_size":296,"log_dups_size":0,"ondisk_log_size":296,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2424832,"num_objects":37,"num_object_clones":0,"num_object_copies":111,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":296,"num_write_kb":2368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.d","version":"93'271","reported_seq":362,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-22T03:15:35.660012+0000","last_change":"2024-09-22T03:15:33.956874+0000","last_active":"2024-09-22T03:15:35.660012+0000","last_peered":"2024-09-22T03:15:35.660012+0000","last_clean":"2024-09-22T03:15:35.660012+0000","last_became_active":"2024-09-22T03:15:33.953772+0000","last_became_peered":"2024-09-22T03:15:33.953772+0000","last_unstale":"2024-09-22T03:15:35.660012+0000","last_undegraded":"2024-09-22T03:15:35.660012+0000","last_fullsized":"2024-09-22T03:15:35.660012+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_clean_scrub_stamp":"2024-09-22T03:15:18.897706+0000","objects_scrubbed":0,"log_size":271,"log_dups_size":0,"ondisk_log_size":271,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2228224,"num_objects":34,"num_object_clones":0,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":272,"num_write_kb":2176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":1835008,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.c","version":"93'199","reported_seq":266,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-22T03:15:35.696026+0000","last_change":"2024-09-22T03:15:33.954730+0000","last_active":"2024-09-22T03:15:35.696026+0000","last_peered":"2024-09-22T03:15:35.696026+0000","last_clean":"2024-09-22T03:15:35.696026+0000","last_became_active":"2024-09-22T03:15:33.951105+0000","last_became_peered":"2024-09-22T03:15:33.951105+0000","last_unstale":"2024-09-22T03:15:35.696026+0000","last_undegraded":"2024-09-22T03:15:35.696026+0000","last_fullsized":"2024-09-22T03:15:35.696026+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_clean_scrub_stamp":"2024-09-22T03:15:18.897706+0000","objects_scrubbed":0,"log_size":199,"log_dups_size":0,"ondisk_log_size":199,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1638400,"num_objects":25,"num_object_clones":0,"num_object_copies":75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":200,"num_write_kb":1600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":1048576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.b","version":"93'360","reported_seq":384,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-22T03:15:38.932563+0000","last_change":"2024-09-22T03:15:34.925809+0000","last_active":"2024-09-22T03:15:38.932563+0000","last_peered":"2024-09-22T03:15:38.932563+0000","last_clean":"2024-09-22T03:15:38.932563+0000","last_became_active":"2024-09-22T03:15:34.920855+0000","last_became_peered":"2024-09-22T03:15:34.920855+0000","last_unstale":"2024-09-22T03:15:38.932563+0000","last_undegraded":"2024-09-22T03:15:38.932563+0000","last_fullsized":"2024-09-22T03:15:38.932563+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_clean_scrub_stamp":"2024-09-22T03:15:18.897706+0000","objects_scrubbed":0,"log_size":360,"log_dups_size":0,"ondisk_log_size":360,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2949120,"num_objects":45,"num_object_clones":0,"num_object_copies":135,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":360,"num_write_kb":2880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":2031616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"94'303","reported_seq":331,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-22T03:15:39.104339+0000","last_change":"2024-09-22T03:15:33.977496+0000","last_active":"2024-09-22T03:15:39.104339+0000","last_peered":"2024-09-22T03:15:39.104339+0000","last_clean":"2024-09-22T03:15:39.104339+0000","last_became_active":"2024-09-22T03:15:33.972140+0000","last_became_peered":"2024-09-22T03:15:33.972140+0000","last_unstale":"2024-09-22T03:15:39.104339+0000","last_undegraded":"2024-09-22T03:15:39.104339+0000","last_fullsized":"2024-09-22T03:15:39.104339+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_clean_scrub_stamp":"2024-09-22T03:15:18.897706+0000","objects_scrubbed":0,"log_size":303,"log_dups_size":0,"ondisk_log_size":303,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2490368,"num_objects":38,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":304,"num_write_kb":2432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":1507328,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.9","version":"93'207","reported_seq":278,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-22T03:15:35.830737+0000","last_change":"2024-09-22T03:15:33.948202+0000","last_active":"2024-09-22T03:15:35.830737+0000","last_peered":"2024-09-22T03:15:35.830737+0000","last_clean":"2024-09-22T03:15:35.830737+0000","last_became_active":"2024-09-22T03:15:33.943275+0000","last_became_peered":"2024-09-22T03:15:33.943275+0000","last_unstale":"2024-09-22T03:15:35.830737+0000","last_undegraded":"2024-09-22T03:15:35.830737+0000","last_fullsized":"2024-09-22T03:15:35.830737+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_clean_scrub_stamp":"2024-09-22T03:15:18.897706+0000","objects_scrubbed":0,"log_size":207,"log_dups_size":0,"ondisk_log_size":207,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1703936,"num_objects":26,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":208,"num_write_kb":1664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":1179648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.8","version":"93'272","reported_seq":355,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-22T03:15:38.940777+0000","last_change":"2024-09-22T03:15:33.946927+0000","last_active":"2024-09-22T03:15:38.940777+0000","last_peered":"2024-09-22T03:15:38.940777+0000","last_clean":"2024-09-22T03:15:38.940777+0000","last_became_active":"2024-09-22T03:15:33.940267+0000","last_became_peered":"2024-09-22T03:15:33.940267+0000","last_unstale":"2024-09-22T03:15:38.940777+0000","last_undegraded":"2024-09-22T03:15:38.940777+0000","last_fullsized":"2024-09-22T03:15:38.940777+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_clean_scrub_stamp":"2024-09-22T03:15:18.897706+0000","objects_scrubbed":0,"log_size":272,"log_dups_size":0,"ondisk_log_size":272,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2228224,"num_objects":34,"num_object_clones":0,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":272,"num_write_kb":2176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":1507328,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.6","version":"93'328","reported_seq":384,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-22T03:15:38.934648+0000","last_change":"2024-09-22T03:15:32.999617+0000","last_active":"2024-09-22T03:15:38.934648+0000","last_peered":"2024-09-22T03:15:38.934648+0000","last_clean":"2024-09-22T03:15:38.934648+0000","last_became_active":"2024-09-22T03:15:32.993508+0000","last_became_peered":"2024-09-22T03:15:32.993508+0000","last_unstale":"2024-09-22T03:15:38.934648+0000","last_undegraded":"2024-09-22T03:15:38.934648+0000","last_fullsized":"2024-09-22T03:15:38.934648+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_clean_scrub_stamp":"2024-09-22T03:15:18.897706+0000","objects_scrubbed":0,"log_size":328,"log_dups_size":0,"ondisk_log_size":328,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2686976,"num_objects":41,"num_object_clones":0,"num_object_copies":123,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":328,"num_write_kb":2624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":491520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":118,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-22T03:15:38.930484+0000","last_change":"2024-09-22T03:15:30.926185+0000","last_active":"2024-09-22T03:15:38.930484+0000","last_peered":"2024-09-22T03:15:38.930484+0000","last_clean":"2024-09-22T03:15:38.930484+0000","last_became_active":"2024-09-22T03:15:29.957045+0000","last_became_peered":"2024-09-22T03:15:29.957045+0000","last_unstale":"2024-09-22T03:15:38.930484+0000","last_undegraded":"2024-09-22T03:15:38.930484+0000","last_fullsized":"2024-09-22T03:15:38.930484+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.7","version":"94'431","reported_seq":525,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-22T03:15:39.133093+0000","last_change":"2024-09-22T03:15:33.949109+0000","last_active":"2024-09-22T03:15:39.133093+0000","last_peered":"2024-09-22T03:15:39.133093+0000","last_clean":"2024-09-22T03:15:39.133093+0000","last_became_active":"2024-09-22T03:15:33.943481+0000","last_became_peered":"2024-09-22T03:15:33.943481+0000","last_unstale":"2024-09-22T03:15:39.133093+0000","last_undegraded":"2024-09-22T03:15:39.133093+0000","last_fullsized":"2024-09-22T03:15:39.133093+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_clean_scrub_stamp":"2024-09-22T03:15:18.897706+0000","objects_scrubbed":0,"log_size":431,"log_dups_size":0,"ondisk_log_size":431,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3538944,"num_objects":54,"num_object_clones":0,"num_object_copies":162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":432,"num_write_kb":3456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":1835008,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":198,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-22T03:15:34.908280+0000","last_change":"2024-09-22T03:14:38.225116+0000","last_active":"2024-09-22T03:15:34.908280+0000","last_peered":"2024-09-22T03:15:34.908280+0000","last_clean":"2024-09-22T03:15:34.908280+0000","last_became_active":"2024-09-22T03:14:37.266115+0000","last_became_peered":"2024-09-22T03:14:37.266115+0000","last_unstale":"2024-09-22T03:15:34.908280+0000","last_undegraded":"2024-09-22T03:15:34.908280+0000","last_fullsized":"2024-09-22T03:15:34.908280+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.4","version":"93'255","reported_seq":332,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-22T03:15:36.014164+0000","last_change":"2024-09-22T03:15:33.941381+0000","last_active":"2024-09-22T03:15:36.014164+0000","last_peered":"2024-09-22T03:15:36.014164+0000","last_clean":"2024-09-22T03:15:36.014164+0000","last_became_active":"2024-09-22T03:15:33.938030+0000","last_became_peered":"2024-09-22T03:15:33.938030+0000","last_unstale":"2024-09-22T03:15:36.014164+0000","last_undegraded":"2024-09-22T03:15:36.014164+0000","last_fullsized":"2024-09-22T03:15:36.014164+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_clean_scrub_stamp":"2024-09-22T03:15:18.897706+0000","objects_scrubbed":0,"log_size":255,"log_dups_size":0,"ondisk_log_size":255,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2097152,"num_objects":32,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":256,"num_write_kb":2048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1376256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":227,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-22T03:15:34.915478+0000","last_change":"2024-09-22T03:15:30.950741+0000","last_active":"2024-09-22T03:15:34.915478+0000","last_peered":"2024-09-22T03:15:34.915478+0000","last_clean":"2024-09-22T03:15:34.915478+0000","last_became_active":"2024-09-22T03:15:29.979306+0000","last_became_peered":"2024-09-22T03:15:29.979306+0000","last_unstale":"2024-09-22T03:15:34.915478+0000","last_undegraded":"2024-09-22T03:15:34.915478+0000","last_fullsized":"2024-09-22T03:15:34.915478+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.5","version":"94'364","reported_seq":463,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-22T03:15:39.184838+0000","last_change":"2024-09-22T03:15:32.978797+0000","last_active":"2024-09-22T03:15:39.184838+0000","last_peered":"2024-09-22T03:15:39.184838+0000","last_clean":"2024-09-22T03:15:39.184838+0000","last_became_active":"2024-09-22T03:15:32.974824+0000","last_became_peered":"2024-09-22T03:15:32.974824+0000","last_unstale":"2024-09-22T03:15:39.184838+0000","last_undegraded":"2024-09-22T03:15:39.184838+0000","last_fullsized":"2024-09-22T03:15:39.184838+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_clean_scrub_stamp":"2024-09-22T03:15:18.897706+0000","objects_scrubbed":0,"log_size":364,"log_dups_size":0,"ondisk_log_size":364,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2990080,"num_objects":46,"num_object_clones":0,"num_object_copies":138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":365,"num_write_kb":2920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":1966080,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":179,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-22T03:15:38.933214+0000","last_change":"2024-09-22T03:15:30.924363+0000","last_active":"2024-09-22T03:15:38.933214+0000","last_peered":"2024-09-22T03:15:38.933214+0000","last_clean":"2024-09-22T03:15:38.933214+0000","last_became_active":"2024-09-22T03:15:29.958549+0000","last_became_peered":"2024-09-22T03:15:29.958549+0000","last_unstale":"2024-09-22T03:15:38.933214+0000","last_undegraded":"2024-09-22T03:15:38.933214+0000","last_fullsized":"2024-09-22T03:15:38.933214+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"12'97","reported_seq":321,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-22T03:15:34.910558+0000","last_change":"2024-09-22T03:14:37.276946+0000","last_active":"2024-09-22T03:15:34.910558+0000","last_peered":"2024-09-22T03:15:34.910558+0000","last_clean":"2024-09-22T03:15:34.910558+0000","last_became_active":"2024-09-22T03:14:37.273858+0000","last_became_peered":"2024-09-22T03:14:37.273858+0000","last_unstale":"2024-09-22T03:15:34.910558+0000","last_undegraded":"2024-09-22T03:15:34.910558+0000","last_fullsized":"2024-09-22T03:15:34.910558+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.2","version":"93'320","reported_seq":371,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-22T03:15:38.931185+0000","last_change":"2024-09-22T03:15:33.953595+0000","last_active":"2024-09-22T03:15:38.931185+0000","last_peered":"2024-09-22T03:15:38.931185+0000","last_clean":"2024-09-22T03:15:38.931185+0000","last_became_active":"2024-09-22T03:15:33.947435+0000","last_became_peered":"2024-09-22T03:15:33.947435+0000","last_unstale":"2024-09-22T03:15:38.931185+0000","last_undegraded":"2024-09-22T03:15:38.931185+0000","last_fullsized":"2024-09-22T03:15:38.931185+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_clean_scrub_stamp":"2024-09-22T03:15:18.897706+0000","objects_scrubbed":0,"log_size":320,"log_dups_size":0,"ondisk_log_size":320,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2621440,"num_objects":40,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":320,"num_write_kb":2560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1376256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":206,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-22T03:15:38.932153+0000","last_change":"2024-09-22T03:14:38.265993+0000","last_active":"2024-09-22T03:15:38.932153+0000","last_peered":"2024-09-22T03:15:38.932153+0000","last_clean":"2024-09-22T03:15:38.932153+0000","last_became_active":"2024-09-22T03:14:37.276817+0000","last_became_peered":"2024-09-22T03:14:37.276817+0000","last_unstale":"2024-09-22T03:15:38.932153+0000","last_undegraded":"2024-09-22T03:15:38.932153+0000","last_fullsized":"2024-09-22T03:15:38.932153+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.3","version":"94'288","reported_seq":336,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-22T03:15:39.077694+0000","last_change":"2024-09-22T03:15:33.961634+0000","last_active":"2024-09-22T03:15:39.077694+0000","last_peered":"2024-09-22T03:15:39.077694+0000","last_clean":"2024-09-22T03:15:39.077694+0000","last_became_active":"2024-09-22T03:15:33.957171+0000","last_became_peered":"2024-09-22T03:15:33.957171+0000","last_unstale":"2024-09-22T03:15:39.077694+0000","last_undegraded":"2024-09-22T03:15:39.077694+0000","last_fullsized":"2024-09-22T03:15:39.077694+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_clean_scrub_stamp":"2024-09-22T03:15:18.897706+0000","objects_scrubbed":0,"log_size":288,"log_dups_size":0,"ondisk_log_size":288,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2367488,"num_objects":37,"num_object_clones":0,"num_object_copies":111,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":289,"num_write_kb":2312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1376256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":206,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-22T03:15:34.915279+0000","last_change":"2024-09-22T03:15:30.950892+0000","last_active":"2024-09-22T03:15:34.915279+0000","last_peered":"2024-09-22T03:15:34.915279+0000","last_clean":"2024-09-22T03:15:34.915279+0000","last_became_active":"2024-09-22T03:15:29.963452+0000","last_became_peered":"2024-09-22T03:15:29.963452+0000","last_unstale":"2024-09-22T03:15:34.915279+0000","last_undegraded":"2024-09-22T03:15:34.915279+0000","last_fullsized":"2024-09-22T03:15:34.915279+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.0","version":"93'224","reported_seq":304,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-22T03:15:35.470189+0000","last_change":"2024-09-22T03:15:32.976320+0000","last_active":"2024-09-22T03:15:35.470189+0000","last_peered":"2024-09-22T03:15:35.470189+0000","last_clean":"2024-09-22T03:15:35.470189+0000","last_became_active":"2024-09-22T03:15:32.973547+0000","last_became_peered":"2024-09-22T03:15:32.973547+0000","last_unstale":"2024-09-22T03:15:35.470189+0000","last_undegraded":"2024-09-22T03:15:35.470189+0000","last_fullsized":"2024-09-22T03:15:35.470189+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_clean_scrub_stamp":"2024-09-22T03:15:18.897706+0000","objects_scrubbed":0,"log_size":224,"log_dups_size":0,"ondisk_log_size":224,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1843200,"num_objects":29,"num_object_clones":0,"num_object_copies":87,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":225,"num_write_kb":1800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1327104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":220,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-22T03:15:38.940364+0000","last_change":"2024-09-22T03:15:30.958044+0000","last_active":"2024-09-22T03:15:38.940364+0000","last_peered":"2024-09-22T03:15:38.940364+0000","last_clean":"2024-09-22T03:15:38.940364+0000","last_became_active":"2024-09-22T03:15:29.976915+0000","last_became_peered":"2024-09-22T03:15:29.976915+0000","last_unstale":"2024-09-22T03:15:38.940364+0000","last_undegraded":"2024-09-22T03:15:38.940364+0000","last_fullsized":"2024-09-22T03:15:38.940364+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1","version":"93'167","reported_seq":230,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-22T03:15:36.448754+0000","last_change":"2024-09-22T03:15:33.950769+0000","last_active":"2024-09-22T03:15:36.448754+0000","last_peered":"2024-09-22T03:15:36.448754+0000","last_clean":"2024-09-22T03:15:36.448754+0000","last_became_active":"2024-09-22T03:15:33.945111+0000","last_became_peered":"2024-09-22T03:15:33.945111+0000","last_unstale":"2024-09-22T03:15:36.448754+0000","last_undegraded":"2024-09-22T03:15:36.448754+0000","last_fullsized":"2024-09-22T03:15:36.448754+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_clean_scrub_stamp":"2024-09-22T03:15:18.897706+0000","objects_scrubbed":0,"log_size":167,"log_dups_size":0,"ondisk_log_size":167,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1376256,"num_objects":21,"num_object_clones":0,"num_object_copies":63,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":168,"num_write_kb":1344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":917504,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":215,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-22T03:15:38.941621+0000","last_change":"2024-09-22T03:15:30.958189+0000","last_active":"2024-09-22T03:15:38.941621+0000","last_peered":"2024-09-22T03:15:38.941621+0000","last_clean":"2024-09-22T03:15:38.941621+0000","last_became_active":"2024-09-22T03:15:29.980027+0000","last_became_peered":"2024-09-22T03:15:29.980027+0000","last_unstale":"2024-09-22T03:15:38.941621+0000","last_undegraded":"2024-09-22T03:15:38.941621+0000","last_fullsized":"2024-09-22T03:15:38.941621+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":37740544,"num_objects":578,"num_object_clones":0,"num_object_copies":1734,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4607,"num_write_kb":36856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":322,"num_bytes_recovered":21020672,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":46374912,"data_stored":112435200,"data_compressed":5715916,"data_compressed_allocated":46374912,"data_compressed_original":112435200,"omap_allocated":0,"internal_metadata":0},"log_size":4596,"ondisk_log_size":4596,"up":48,"acting":48,"num_store_stats":4},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1180736,"data_compressed":59924,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":87,"seq":373662154762,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":51008,"kb_used_data":9392,"kb_used_omap":0,"kb_used_meta":41600,"kb_avail":94320832,"statfs":{"total":96636764160,"available":96584531968,"internally_reserved":0,"allocated":9617408,"data_stored":37612984,"data_compressed":1838618,"data_compressed_allocated":8822784,"data_compressed_original":37298176,"omap_allocated":0,"internal_metadata":42598400},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":71,"seq":304942678091,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":45536,"kb_used_data":9424,"kb_used_omap":0,"kb_used_meta":36096,"kb_avail":94326304,"statfs":{"total":96636764160,"available":96590135296,"internally_reserved":0,"allocated":9650176,"data_stored":18506519,"data_compressed":938351,"data_compressed_allocated":8843264,"data_compressed_original":18178048,"omap_allocated":0,"internal_metadata":36962304},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":53,"seq":227633266872,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":48948,"kb_used_data":16740,"kb_used_omap":0,"kb_used_meta":32192,"kb_avail":94322892,"statfs":{"total":96636764160,"available":96586641408,"internally_reserved":0,"allocated":17141760,"data_stored":33484760,"data_compressed":1711330,"data_compressed_allocated":16326656,"data_compressed_original":33153024,"omap_allocated":0,"internal_metadata":32964608},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":73,"seq":313532612668,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":48264,"kb_used_data":11960,"kb_used_omap":0,"kb_used_meta":36288,"kb_avail":94323576,"statfs":{"total":96636764160,"available":96587341824,"internally_reserved":0,"allocated":12247040,"data_stored":23732996,"data_compressed":1207946,"data_compressed_allocated":11444224,"data_compressed_original":23404544,"omap_allocated":0,"internal_metadata":37158912},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29962,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29962,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":11980800,"data_stored":23986176,"data_compressed":1237627,"data_compressed_allocated":11980800,"data_compressed_original":23986176,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16461824,"data_stored":32931840,"data_compressed":1699557,"data_compressed_allocated":16461824,"data_compressed_original":32931840,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8867840,"data_stored":17735680,"data_compressed":915157,"data_compressed_allocated":8867840,"data_compressed_original":17735680,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":9064448,"data_stored":37781504,"data_compressed":1863575,"data_compressed_allocated":9064448,"data_compressed_original":37781504,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-22T03:15:41.981 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 21 16 5839 5823 2.16611 2.35156 0.0762327 0.057625 2024-09-22T03:15:42.981 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 22 16 6192 6176 2.193 2.75781 0.0467399 0.0569112 2024-09-22T03:15:43.256 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-22T03:15:43.389 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:15:43.385+0000 7fbaa070e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:15:43.392 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:15:43.391+0000 7fbaa070e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:15:43.981 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 23 16 6528 6512 2.21177 2.625 0.0333908 0.0564679 2024-09-22T03:15:44.981 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 24 16 6884 6868 2.23549 2.78125 0.0335651 0.0558475 2024-09-22T03:15:45.580 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-22T03:15:45.697 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:15:45.693+0000 7f3394597640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:15:45.701 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:15:45.700+0000 7f3394597640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:15:45.981 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 25 16 7225 7209 2.25263 2.66406 0.0329136 0.0554377 2024-09-22T03:15:46.683 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [1, 2, 0, 3] 2024-09-22T03:15:46.683 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-09-22T03:15:46.827 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:15:46.818+0000 7f0d96797640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:15:46.831 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:15:46.828+0000 7f0d96797640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:15:46.981 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 26 16 7523 7507 2.25552 2.32812 0.0448361 0.0553526 2024-09-22T03:15:47.866 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-22T03:15:47.958 INFO:teuthology.orchestra.run.smithi153.stderr:reweighted osd.0 to 1 (10000) 2024-09-22T03:15:47.975 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-09-22T03:15:47.982 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 27 16 7846 7830 2.26544 2.52344 0.0980034 0.0550684 2024-09-22T03:15:48.037 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:15:48.033+0000 7fe2b3c58640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:15:48.042 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:15:48.039+0000 7fe2b3c58640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:15:48.132 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:15:48.118+0000 7f59c3597640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:15:48.138 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:15:48.136+0000 7f59c3597640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:15:48.905 INFO:teuthology.orchestra.run.smithi153.stderr:reweighted osd.1 to 1 (10000) 2024-09-22T03:15:48.933 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-09-22T03:15:48.982 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 28 15 8155 8140 2.27102 2.42188 0.0817171 0.0549722 2024-09-22T03:15:49.058 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:15:49.050+0000 7f4370e6c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:15:49.062 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:15:49.060+0000 7f4370e6c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:15:49.975 INFO:teuthology.orchestra.run.smithi153.stderr:reweighted osd.2 to 1 (10000) 2024-09-22T03:15:49.982 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 29 16 8466 8450 2.27621 2.42188 0.0451011 0.0548491 2024-09-22T03:15:49.995 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-09-22T03:15:50.136 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:15:50.127+0000 7f13ae797640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:15:50.140 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:15:50.137+0000 7f13ae797640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:15:50.282 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-22T03:15:50.429 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:15:50.424+0000 7fc8a8023640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:15:50.433 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:15:50.430+0000 7fc8a8023640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:15:50.980 INFO:teuthology.orchestra.run.smithi153.stderr:reweighted osd.3 to 1 (10000) 2024-09-22T03:15:50.982 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 30 16 8785 8769 2.2834 2.49219 0.03695 0.054667 2024-09-22T03:15:50.996 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-09-22T03:15:50.996 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-22T03:15:51.136 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:15:51.128+0000 7f5cd046c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:15:51.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:15:51.138+0000 7f5cd046c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:15:51.403 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-22T03:15:51.403 INFO:teuthology.orchestra.run.smithi153.stderr:dumped all 2024-09-22T03:15:51.418 INFO:teuthology.orchestra.run.smithi153.stdout:{"pg_ready":true,"pg_map":{"version":325,"stamp":"2024-09-22T03:15:50.982061+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":64741939,"num_objects":984,"num_object_clones":0,"num_object_copies":2952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":7970,"num_write_kb":64544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":322,"num_bytes_recovered":21020672,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7921,"ondisk_log_size":7921,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":220912,"kb_used_data":87152,"kb_used_omap":0,"kb_used_meta":133696,"kb_avail":377266448,"statfs":{"total":386547056640,"available":386320842752,"internally_reserved":0,"allocated":89243648,"data_stored":194433105,"data_compressed":9882676,"data_compressed_allocated":85975040,"data_compressed_original":193110016,"omap_allocated":0,"internal_metadata":136904704},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":8658944,"num_objects":132,"num_object_clones":0,"num_object_copies":396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1057,"num_write_kb":8456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.775135"},"pg_stats":[{"pgid":"7.f","version":"95'560","reported_seq":611,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-22T03:15:48.920863+0000","last_change":"2024-09-22T03:15:33.972925+0000","last_active":"2024-09-22T03:15:48.920863+0000","last_peered":"2024-09-22T03:15:48.920863+0000","last_clean":"2024-09-22T03:15:48.920863+0000","last_became_active":"2024-09-22T03:15:33.967199+0000","last_became_peered":"2024-09-22T03:15:33.967199+0000","last_unstale":"2024-09-22T03:15:48.920863+0000","last_undegraded":"2024-09-22T03:15:48.920863+0000","last_fullsized":"2024-09-22T03:15:48.920863+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_clean_scrub_stamp":"2024-09-22T03:15:18.897706+0000","objects_scrubbed":0,"log_size":560,"log_dups_size":0,"ondisk_log_size":560,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4587520,"num_objects":70,"num_object_clones":0,"num_object_copies":210,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":560,"num_write_kb":4480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":1245184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.e","version":"96'487","reported_seq":530,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-22T03:15:48.957583+0000","last_change":"2024-09-22T03:15:20.007094+0000","last_active":"2024-09-22T03:15:48.957583+0000","last_peered":"2024-09-22T03:15:48.957583+0000","last_clean":"2024-09-22T03:15:48.957583+0000","last_became_active":"2024-09-22T03:15:20.000757+0000","last_became_peered":"2024-09-22T03:15:20.000757+0000","last_unstale":"2024-09-22T03:15:48.957583+0000","last_undegraded":"2024-09-22T03:15:48.957583+0000","last_fullsized":"2024-09-22T03:15:48.957583+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_clean_scrub_stamp":"2024-09-22T03:15:18.897706+0000","objects_scrubbed":0,"log_size":487,"log_dups_size":0,"ondisk_log_size":487,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3997696,"num_objects":61,"num_object_clones":0,"num_object_copies":183,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":488,"num_write_kb":3904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.d","version":"94'511","reported_seq":605,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-22T03:15:45.635552+0000","last_change":"2024-09-22T03:15:33.956874+0000","last_active":"2024-09-22T03:15:45.635552+0000","last_peered":"2024-09-22T03:15:45.635552+0000","last_clean":"2024-09-22T03:15:45.635552+0000","last_became_active":"2024-09-22T03:15:33.953772+0000","last_became_peered":"2024-09-22T03:15:33.953772+0000","last_unstale":"2024-09-22T03:15:45.635552+0000","last_undegraded":"2024-09-22T03:15:45.635552+0000","last_fullsized":"2024-09-22T03:15:45.635552+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_clean_scrub_stamp":"2024-09-22T03:15:18.897706+0000","objects_scrubbed":0,"log_size":511,"log_dups_size":0,"ondisk_log_size":511,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":64,"num_object_clones":0,"num_object_copies":192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":512,"num_write_kb":4096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":1835008,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.c","version":"94'415","reported_seq":485,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-22T03:15:45.971334+0000","last_change":"2024-09-22T03:15:33.954730+0000","last_active":"2024-09-22T03:15:45.971334+0000","last_peered":"2024-09-22T03:15:45.971334+0000","last_clean":"2024-09-22T03:15:45.971334+0000","last_became_active":"2024-09-22T03:15:33.951105+0000","last_became_peered":"2024-09-22T03:15:33.951105+0000","last_unstale":"2024-09-22T03:15:45.971334+0000","last_undegraded":"2024-09-22T03:15:45.971334+0000","last_fullsized":"2024-09-22T03:15:45.971334+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_clean_scrub_stamp":"2024-09-22T03:15:18.897706+0000","objects_scrubbed":0,"log_size":415,"log_dups_size":0,"ondisk_log_size":415,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3407872,"num_objects":52,"num_object_clones":0,"num_object_copies":156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":416,"num_write_kb":3328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":1048576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.b","version":"95'560","reported_seq":590,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-22T03:15:48.924375+0000","last_change":"2024-09-22T03:15:34.925809+0000","last_active":"2024-09-22T03:15:48.924375+0000","last_peered":"2024-09-22T03:15:48.924375+0000","last_clean":"2024-09-22T03:15:48.924375+0000","last_became_active":"2024-09-22T03:15:34.920855+0000","last_became_peered":"2024-09-22T03:15:34.920855+0000","last_unstale":"2024-09-22T03:15:48.924375+0000","last_undegraded":"2024-09-22T03:15:48.924375+0000","last_fullsized":"2024-09-22T03:15:48.924375+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_clean_scrub_stamp":"2024-09-22T03:15:18.897706+0000","objects_scrubbed":0,"log_size":560,"log_dups_size":0,"ondisk_log_size":560,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4587520,"num_objects":70,"num_object_clones":0,"num_object_copies":210,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":560,"num_write_kb":4480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":2031616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"95'456","reported_seq":489,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-22T03:15:48.923603+0000","last_change":"2024-09-22T03:15:33.977496+0000","last_active":"2024-09-22T03:15:48.923603+0000","last_peered":"2024-09-22T03:15:48.923603+0000","last_clean":"2024-09-22T03:15:48.923603+0000","last_became_active":"2024-09-22T03:15:33.972140+0000","last_became_peered":"2024-09-22T03:15:33.972140+0000","last_unstale":"2024-09-22T03:15:48.923603+0000","last_undegraded":"2024-09-22T03:15:48.923603+0000","last_fullsized":"2024-09-22T03:15:48.923603+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_clean_scrub_stamp":"2024-09-22T03:15:18.897706+0000","objects_scrubbed":0,"log_size":456,"log_dups_size":0,"ondisk_log_size":456,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3735552,"num_objects":57,"num_object_clones":0,"num_object_copies":171,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":456,"num_write_kb":3648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":1507328,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.9","version":"94'399","reported_seq":473,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-22T03:15:45.585789+0000","last_change":"2024-09-22T03:15:33.948202+0000","last_active":"2024-09-22T03:15:45.585789+0000","last_peered":"2024-09-22T03:15:45.585789+0000","last_clean":"2024-09-22T03:15:45.585789+0000","last_became_active":"2024-09-22T03:15:33.943275+0000","last_became_peered":"2024-09-22T03:15:33.943275+0000","last_unstale":"2024-09-22T03:15:45.585789+0000","last_undegraded":"2024-09-22T03:15:45.585789+0000","last_fullsized":"2024-09-22T03:15:45.585789+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_clean_scrub_stamp":"2024-09-22T03:15:18.897706+0000","objects_scrubbed":0,"log_size":399,"log_dups_size":0,"ondisk_log_size":399,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3276800,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":400,"num_write_kb":3200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":1179648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.8","version":"95'472","reported_seq":561,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-22T03:15:48.931253+0000","last_change":"2024-09-22T03:15:33.946927+0000","last_active":"2024-09-22T03:15:48.931253+0000","last_peered":"2024-09-22T03:15:48.931253+0000","last_clean":"2024-09-22T03:15:48.931253+0000","last_became_active":"2024-09-22T03:15:33.940267+0000","last_became_peered":"2024-09-22T03:15:33.940267+0000","last_unstale":"2024-09-22T03:15:48.931253+0000","last_undegraded":"2024-09-22T03:15:48.931253+0000","last_fullsized":"2024-09-22T03:15:48.931253+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_clean_scrub_stamp":"2024-09-22T03:15:18.897706+0000","objects_scrubbed":0,"log_size":472,"log_dups_size":0,"ondisk_log_size":472,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3866624,"num_objects":59,"num_object_clones":0,"num_object_copies":177,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":472,"num_write_kb":3776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":1507328,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.6","version":"95'560","reported_seq":622,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-22T03:15:48.922886+0000","last_change":"2024-09-22T03:15:32.999617+0000","last_active":"2024-09-22T03:15:48.922886+0000","last_peered":"2024-09-22T03:15:48.922886+0000","last_clean":"2024-09-22T03:15:48.922886+0000","last_became_active":"2024-09-22T03:15:32.993508+0000","last_became_peered":"2024-09-22T03:15:32.993508+0000","last_unstale":"2024-09-22T03:15:48.922886+0000","last_undegraded":"2024-09-22T03:15:48.922886+0000","last_fullsized":"2024-09-22T03:15:48.922886+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_clean_scrub_stamp":"2024-09-22T03:15:18.897706+0000","objects_scrubbed":0,"log_size":560,"log_dups_size":0,"ondisk_log_size":560,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4587520,"num_objects":70,"num_object_clones":0,"num_object_copies":210,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":560,"num_write_kb":4480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":491520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":124,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-22T03:15:48.924682+0000","last_change":"2024-09-22T03:15:30.926185+0000","last_active":"2024-09-22T03:15:48.924682+0000","last_peered":"2024-09-22T03:15:48.924682+0000","last_clean":"2024-09-22T03:15:48.924682+0000","last_became_active":"2024-09-22T03:15:29.957045+0000","last_became_peered":"2024-09-22T03:15:29.957045+0000","last_unstale":"2024-09-22T03:15:48.924682+0000","last_undegraded":"2024-09-22T03:15:48.924682+0000","last_fullsized":"2024-09-22T03:15:48.924682+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.7","version":"96'639","reported_seq":739,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-22T03:15:49.099772+0000","last_change":"2024-09-22T03:15:33.949109+0000","last_active":"2024-09-22T03:15:49.099772+0000","last_peered":"2024-09-22T03:15:49.099772+0000","last_clean":"2024-09-22T03:15:49.099772+0000","last_became_active":"2024-09-22T03:15:33.943481+0000","last_became_peered":"2024-09-22T03:15:33.943481+0000","last_unstale":"2024-09-22T03:15:49.099772+0000","last_undegraded":"2024-09-22T03:15:49.099772+0000","last_fullsized":"2024-09-22T03:15:49.099772+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_clean_scrub_stamp":"2024-09-22T03:15:18.897706+0000","objects_scrubbed":0,"log_size":639,"log_dups_size":0,"ondisk_log_size":639,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5242880,"num_objects":80,"num_object_clones":0,"num_object_copies":240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":640,"num_write_kb":5120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":1835008,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":201,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-22T03:15:38.911987+0000","last_change":"2024-09-22T03:14:38.225116+0000","last_active":"2024-09-22T03:15:38.911987+0000","last_peered":"2024-09-22T03:15:38.911987+0000","last_clean":"2024-09-22T03:15:38.911987+0000","last_became_active":"2024-09-22T03:14:37.266115+0000","last_became_peered":"2024-09-22T03:14:37.266115+0000","last_unstale":"2024-09-22T03:15:38.911987+0000","last_undegraded":"2024-09-22T03:15:38.911987+0000","last_fullsized":"2024-09-22T03:15:38.911987+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.4","version":"94'455","reported_seq":535,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-22T03:15:45.904412+0000","last_change":"2024-09-22T03:15:33.941381+0000","last_active":"2024-09-22T03:15:45.904412+0000","last_peered":"2024-09-22T03:15:45.904412+0000","last_clean":"2024-09-22T03:15:45.904412+0000","last_became_active":"2024-09-22T03:15:33.938030+0000","last_became_peered":"2024-09-22T03:15:33.938030+0000","last_unstale":"2024-09-22T03:15:45.904412+0000","last_undegraded":"2024-09-22T03:15:45.904412+0000","last_fullsized":"2024-09-22T03:15:45.904412+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_clean_scrub_stamp":"2024-09-22T03:15:18.897706+0000","objects_scrubbed":0,"log_size":455,"log_dups_size":0,"ondisk_log_size":455,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3735552,"num_objects":57,"num_object_clones":0,"num_object_copies":171,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":456,"num_write_kb":3648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1376256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":230,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-22T03:15:38.909203+0000","last_change":"2024-09-22T03:15:30.950741+0000","last_active":"2024-09-22T03:15:38.909203+0000","last_peered":"2024-09-22T03:15:38.909203+0000","last_clean":"2024-09-22T03:15:38.909203+0000","last_became_active":"2024-09-22T03:15:29.979306+0000","last_became_peered":"2024-09-22T03:15:29.979306+0000","last_unstale":"2024-09-22T03:15:38.909203+0000","last_undegraded":"2024-09-22T03:15:38.909203+0000","last_fullsized":"2024-09-22T03:15:38.909203+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.5","version":"96'538","reported_seq":643,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-22T03:15:49.190464+0000","last_change":"2024-09-22T03:15:32.978797+0000","last_active":"2024-09-22T03:15:49.190464+0000","last_peered":"2024-09-22T03:15:49.190464+0000","last_clean":"2024-09-22T03:15:49.190464+0000","last_became_active":"2024-09-22T03:15:32.974824+0000","last_became_peered":"2024-09-22T03:15:32.974824+0000","last_unstale":"2024-09-22T03:15:49.190464+0000","last_undegraded":"2024-09-22T03:15:49.190464+0000","last_fullsized":"2024-09-22T03:15:49.190464+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_clean_scrub_stamp":"2024-09-22T03:15:18.897706+0000","objects_scrubbed":0,"log_size":538,"log_dups_size":0,"ondisk_log_size":538,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4415488,"num_objects":68,"num_object_clones":0,"num_object_copies":204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":539,"num_write_kb":4312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":1966080,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":185,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-22T03:15:48.921686+0000","last_change":"2024-09-22T03:15:30.924363+0000","last_active":"2024-09-22T03:15:48.921686+0000","last_peered":"2024-09-22T03:15:48.921686+0000","last_clean":"2024-09-22T03:15:48.921686+0000","last_became_active":"2024-09-22T03:15:29.958549+0000","last_became_peered":"2024-09-22T03:15:29.958549+0000","last_unstale":"2024-09-22T03:15:48.921686+0000","last_undegraded":"2024-09-22T03:15:48.921686+0000","last_fullsized":"2024-09-22T03:15:48.921686+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"12'97","reported_seq":324,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-22T03:15:38.915351+0000","last_change":"2024-09-22T03:14:37.276946+0000","last_active":"2024-09-22T03:15:38.915351+0000","last_peered":"2024-09-22T03:15:38.915351+0000","last_clean":"2024-09-22T03:15:38.915351+0000","last_became_active":"2024-09-22T03:14:37.273858+0000","last_became_peered":"2024-09-22T03:14:37.273858+0000","last_unstale":"2024-09-22T03:15:38.915351+0000","last_undegraded":"2024-09-22T03:15:38.915351+0000","last_fullsized":"2024-09-22T03:15:38.915351+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.2","version":"95'560","reported_seq":617,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-22T03:15:48.919500+0000","last_change":"2024-09-22T03:15:33.953595+0000","last_active":"2024-09-22T03:15:48.919500+0000","last_peered":"2024-09-22T03:15:48.919500+0000","last_clean":"2024-09-22T03:15:48.919500+0000","last_became_active":"2024-09-22T03:15:33.947435+0000","last_became_peered":"2024-09-22T03:15:33.947435+0000","last_unstale":"2024-09-22T03:15:48.919500+0000","last_undegraded":"2024-09-22T03:15:48.919500+0000","last_fullsized":"2024-09-22T03:15:48.919500+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_clean_scrub_stamp":"2024-09-22T03:15:18.897706+0000","objects_scrubbed":0,"log_size":560,"log_dups_size":0,"ondisk_log_size":560,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4587520,"num_objects":70,"num_object_clones":0,"num_object_copies":210,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":560,"num_write_kb":4480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1376256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":212,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-22T03:15:48.925906+0000","last_change":"2024-09-22T03:14:38.265993+0000","last_active":"2024-09-22T03:15:48.925906+0000","last_peered":"2024-09-22T03:15:48.925906+0000","last_clean":"2024-09-22T03:15:48.925906+0000","last_became_active":"2024-09-22T03:14:37.276817+0000","last_became_peered":"2024-09-22T03:14:37.276817+0000","last_unstale":"2024-09-22T03:15:48.925906+0000","last_undegraded":"2024-09-22T03:15:48.925906+0000","last_fullsized":"2024-09-22T03:15:48.925906+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.3","version":"96'475","reported_seq":529,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-22T03:15:49.097411+0000","last_change":"2024-09-22T03:15:33.961634+0000","last_active":"2024-09-22T03:15:49.097411+0000","last_peered":"2024-09-22T03:15:49.097411+0000","last_clean":"2024-09-22T03:15:49.097411+0000","last_became_active":"2024-09-22T03:15:33.957171+0000","last_became_peered":"2024-09-22T03:15:33.957171+0000","last_unstale":"2024-09-22T03:15:49.097411+0000","last_undegraded":"2024-09-22T03:15:49.097411+0000","last_fullsized":"2024-09-22T03:15:49.097411+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_clean_scrub_stamp":"2024-09-22T03:15:18.897706+0000","objects_scrubbed":0,"log_size":475,"log_dups_size":0,"ondisk_log_size":475,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3899392,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":476,"num_write_kb":3808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1376256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":209,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-22T03:15:38.908795+0000","last_change":"2024-09-22T03:15:30.950892+0000","last_active":"2024-09-22T03:15:38.908795+0000","last_peered":"2024-09-22T03:15:38.908795+0000","last_clean":"2024-09-22T03:15:38.908795+0000","last_became_active":"2024-09-22T03:15:29.963452+0000","last_became_peered":"2024-09-22T03:15:29.963452+0000","last_unstale":"2024-09-22T03:15:38.908795+0000","last_undegraded":"2024-09-22T03:15:38.908795+0000","last_fullsized":"2024-09-22T03:15:38.908795+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.0","version":"94'391","reported_seq":474,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-22T03:15:45.637023+0000","last_change":"2024-09-22T03:15:32.976320+0000","last_active":"2024-09-22T03:15:45.637023+0000","last_peered":"2024-09-22T03:15:45.637023+0000","last_clean":"2024-09-22T03:15:45.637023+0000","last_became_active":"2024-09-22T03:15:32.973547+0000","last_became_peered":"2024-09-22T03:15:32.973547+0000","last_unstale":"2024-09-22T03:15:45.637023+0000","last_undegraded":"2024-09-22T03:15:45.637023+0000","last_fullsized":"2024-09-22T03:15:45.637023+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_clean_scrub_stamp":"2024-09-22T03:15:18.897706+0000","objects_scrubbed":0,"log_size":391,"log_dups_size":0,"ondisk_log_size":391,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3211264,"num_objects":49,"num_object_clones":0,"num_object_copies":147,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":392,"num_write_kb":3136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1327104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":226,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-22T03:15:48.926975+0000","last_change":"2024-09-22T03:15:30.958044+0000","last_active":"2024-09-22T03:15:48.926975+0000","last_peered":"2024-09-22T03:15:48.926975+0000","last_clean":"2024-09-22T03:15:48.926975+0000","last_became_active":"2024-09-22T03:15:29.976915+0000","last_became_peered":"2024-09-22T03:15:29.976915+0000","last_unstale":"2024-09-22T03:15:48.926975+0000","last_undegraded":"2024-09-22T03:15:48.926975+0000","last_fullsized":"2024-09-22T03:15:48.926975+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1","version":"94'343","reported_seq":409,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-22T03:15:46.859362+0000","last_change":"2024-09-22T03:15:33.950769+0000","last_active":"2024-09-22T03:15:46.859362+0000","last_peered":"2024-09-22T03:15:46.859362+0000","last_clean":"2024-09-22T03:15:46.859362+0000","last_became_active":"2024-09-22T03:15:33.945111+0000","last_became_peered":"2024-09-22T03:15:33.945111+0000","last_unstale":"2024-09-22T03:15:46.859362+0000","last_undegraded":"2024-09-22T03:15:46.859362+0000","last_fullsized":"2024-09-22T03:15:46.859362+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_clean_scrub_stamp":"2024-09-22T03:15:18.897706+0000","objects_scrubbed":0,"log_size":343,"log_dups_size":0,"ondisk_log_size":343,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2818048,"num_objects":43,"num_object_clones":0,"num_object_copies":129,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":344,"num_write_kb":2752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":917504,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":221,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-22T03:15:48.927501+0000","last_change":"2024-09-22T03:15:30.958189+0000","last_active":"2024-09-22T03:15:48.927501+0000","last_peered":"2024-09-22T03:15:48.927501+0000","last_clean":"2024-09-22T03:15:48.927501+0000","last_became_active":"2024-09-22T03:15:29.980027+0000","last_became_peered":"2024-09-22T03:15:29.980027+0000","last_unstale":"2024-09-22T03:15:48.927501+0000","last_undegraded":"2024-09-22T03:15:48.927501+0000","last_fullsized":"2024-09-22T03:15:48.927501+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":64151552,"num_objects":980,"num_object_clones":0,"num_object_copies":2940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7831,"num_write_kb":62648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":322,"num_bytes_recovered":21020672,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":86618112,"data_stored":192921600,"data_compressed":9871891,"data_compressed_allocated":86618112,"data_compressed_original":192921600,"omap_allocated":0,"internal_metadata":0},"log_size":7821,"ondisk_log_size":7821,"up":48,"acting":48,"num_store_stats":4},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1180736,"data_compressed":59924,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":87,"seq":373662154772,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":59256,"kb_used_data":21800,"kb_used_omap":0,"kb_used_meta":37440,"kb_avail":94312584,"statfs":{"total":96636764160,"available":96576086016,"internally_reserved":0,"allocated":22323200,"data_stored":62998346,"data_compressed":3149072,"data_compressed_allocated":21512192,"data_compressed_original":62676992,"omap_allocated":0,"internal_metadata":38338560},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":71,"seq":304942678101,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":44244,"kb_used_data":16452,"kb_used_omap":0,"kb_used_meta":27776,"kb_avail":94327596,"statfs":{"total":96636764160,"available":96591458304,"internally_reserved":0,"allocated":16846848,"data_stored":32886744,"data_compressed":1680716,"data_compressed_allocated":16031744,"data_compressed_original":32555008,"omap_allocated":0,"internal_metadata":28442624},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":53,"seq":227633266882,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":68208,"kb_used_data":27680,"kb_used_omap":0,"kb_used_meta":40512,"kb_avail":94303632,"statfs":{"total":96636764160,"available":96566919168,"internally_reserved":0,"allocated":28344320,"data_stored":55863658,"data_compressed":2866543,"data_compressed_allocated":27512832,"data_compressed_original":55525376,"omap_allocated":0,"internal_metadata":41484288},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":73,"seq":313532612678,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":49204,"kb_used_data":21220,"kb_used_omap":0,"kb_used_meta":27968,"kb_avail":94322636,"statfs":{"total":96636764160,"available":96586379264,"internally_reserved":0,"allocated":21729280,"data_stored":42684357,"data_compressed":2186345,"data_compressed_allocated":20918272,"data_compressed_original":42352640,"omap_allocated":0,"internal_metadata":28639232},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29962,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29962,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":21311488,"data_stored":42647552,"data_compressed":2201221,"data_compressed_allocated":21311488,"data_compressed_original":42647552,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":27643904,"data_stored":55296000,"data_compressed":2854347,"data_compressed_allocated":27643904,"data_compressed_original":55296000,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":15990784,"data_stored":31981568,"data_compressed":1650754,"data_compressed_allocated":15990784,"data_compressed_original":31981568,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":21671936,"data_stored":62996480,"data_compressed":3165569,"data_compressed_allocated":21671936,"data_compressed_original":62996480,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-22T03:15:51.420 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-22T03:15:51.564 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:15:51.557+0000 7f556d26c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:15:51.568 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:15:51.566+0000 7f556d26c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:15:51.812 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-22T03:15:51.812 INFO:teuthology.orchestra.run.smithi153.stderr:dumped all 2024-09-22T03:15:51.834 INFO:teuthology.orchestra.run.smithi153.stdout:{"pg_ready":true,"pg_map":{"version":325,"stamp":"2024-09-22T03:15:50.982061+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":64741939,"num_objects":984,"num_object_clones":0,"num_object_copies":2952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":7970,"num_write_kb":64544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":322,"num_bytes_recovered":21020672,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7921,"ondisk_log_size":7921,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":220912,"kb_used_data":87152,"kb_used_omap":0,"kb_used_meta":133696,"kb_avail":377266448,"statfs":{"total":386547056640,"available":386320842752,"internally_reserved":0,"allocated":89243648,"data_stored":194433105,"data_compressed":9882676,"data_compressed_allocated":85975040,"data_compressed_original":193110016,"omap_allocated":0,"internal_metadata":136904704},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":8658944,"num_objects":132,"num_object_clones":0,"num_object_copies":396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1057,"num_write_kb":8456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.775135"},"pg_stats":[{"pgid":"7.f","version":"95'560","reported_seq":611,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-22T03:15:48.920863+0000","last_change":"2024-09-22T03:15:33.972925+0000","last_active":"2024-09-22T03:15:48.920863+0000","last_peered":"2024-09-22T03:15:48.920863+0000","last_clean":"2024-09-22T03:15:48.920863+0000","last_became_active":"2024-09-22T03:15:33.967199+0000","last_became_peered":"2024-09-22T03:15:33.967199+0000","last_unstale":"2024-09-22T03:15:48.920863+0000","last_undegraded":"2024-09-22T03:15:48.920863+0000","last_fullsized":"2024-09-22T03:15:48.920863+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_clean_scrub_stamp":"2024-09-22T03:15:18.897706+0000","objects_scrubbed":0,"log_size":560,"log_dups_size":0,"ondisk_log_size":560,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4587520,"num_objects":70,"num_object_clones":0,"num_object_copies":210,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":560,"num_write_kb":4480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":1245184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.e","version":"96'487","reported_seq":530,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-22T03:15:48.957583+0000","last_change":"2024-09-22T03:15:20.007094+0000","last_active":"2024-09-22T03:15:48.957583+0000","last_peered":"2024-09-22T03:15:48.957583+0000","last_clean":"2024-09-22T03:15:48.957583+0000","last_became_active":"2024-09-22T03:15:20.000757+0000","last_became_peered":"2024-09-22T03:15:20.000757+0000","last_unstale":"2024-09-22T03:15:48.957583+0000","last_undegraded":"2024-09-22T03:15:48.957583+0000","last_fullsized":"2024-09-22T03:15:48.957583+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_clean_scrub_stamp":"2024-09-22T03:15:18.897706+0000","objects_scrubbed":0,"log_size":487,"log_dups_size":0,"ondisk_log_size":487,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3997696,"num_objects":61,"num_object_clones":0,"num_object_copies":183,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":488,"num_write_kb":3904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.d","version":"94'511","reported_seq":605,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-22T03:15:45.635552+0000","last_change":"2024-09-22T03:15:33.956874+0000","last_active":"2024-09-22T03:15:45.635552+0000","last_peered":"2024-09-22T03:15:45.635552+0000","last_clean":"2024-09-22T03:15:45.635552+0000","last_became_active":"2024-09-22T03:15:33.953772+0000","last_became_peered":"2024-09-22T03:15:33.953772+0000","last_unstale":"2024-09-22T03:15:45.635552+0000","last_undegraded":"2024-09-22T03:15:45.635552+0000","last_fullsized":"2024-09-22T03:15:45.635552+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_clean_scrub_stamp":"2024-09-22T03:15:18.897706+0000","objects_scrubbed":0,"log_size":511,"log_dups_size":0,"ondisk_log_size":511,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":64,"num_object_clones":0,"num_object_copies":192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":512,"num_write_kb":4096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":1835008,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.c","version":"94'415","reported_seq":485,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-22T03:15:45.971334+0000","last_change":"2024-09-22T03:15:33.954730+0000","last_active":"2024-09-22T03:15:45.971334+0000","last_peered":"2024-09-22T03:15:45.971334+0000","last_clean":"2024-09-22T03:15:45.971334+0000","last_became_active":"2024-09-22T03:15:33.951105+0000","last_became_peered":"2024-09-22T03:15:33.951105+0000","last_unstale":"2024-09-22T03:15:45.971334+0000","last_undegraded":"2024-09-22T03:15:45.971334+0000","last_fullsized":"2024-09-22T03:15:45.971334+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_clean_scrub_stamp":"2024-09-22T03:15:18.897706+0000","objects_scrubbed":0,"log_size":415,"log_dups_size":0,"ondisk_log_size":415,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3407872,"num_objects":52,"num_object_clones":0,"num_object_copies":156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":416,"num_write_kb":3328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":1048576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.b","version":"95'560","reported_seq":590,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-22T03:15:48.924375+0000","last_change":"2024-09-22T03:15:34.925809+0000","last_active":"2024-09-22T03:15:48.924375+0000","last_peered":"2024-09-22T03:15:48.924375+0000","last_clean":"2024-09-22T03:15:48.924375+0000","last_became_active":"2024-09-22T03:15:34.920855+0000","last_became_peered":"2024-09-22T03:15:34.920855+0000","last_unstale":"2024-09-22T03:15:48.924375+0000","last_undegraded":"2024-09-22T03:15:48.924375+0000","last_fullsized":"2024-09-22T03:15:48.924375+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_clean_scrub_stamp":"2024-09-22T03:15:18.897706+0000","objects_scrubbed":0,"log_size":560,"log_dups_size":0,"ondisk_log_size":560,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4587520,"num_objects":70,"num_object_clones":0,"num_object_copies":210,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":560,"num_write_kb":4480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":2031616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"95'456","reported_seq":489,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-22T03:15:48.923603+0000","last_change":"2024-09-22T03:15:33.977496+0000","last_active":"2024-09-22T03:15:48.923603+0000","last_peered":"2024-09-22T03:15:48.923603+0000","last_clean":"2024-09-22T03:15:48.923603+0000","last_became_active":"2024-09-22T03:15:33.972140+0000","last_became_peered":"2024-09-22T03:15:33.972140+0000","last_unstale":"2024-09-22T03:15:48.923603+0000","last_undegraded":"2024-09-22T03:15:48.923603+0000","last_fullsized":"2024-09-22T03:15:48.923603+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_clean_scrub_stamp":"2024-09-22T03:15:18.897706+0000","objects_scrubbed":0,"log_size":456,"log_dups_size":0,"ondisk_log_size":456,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3735552,"num_objects":57,"num_object_clones":0,"num_object_copies":171,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":456,"num_write_kb":3648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":1507328,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.9","version":"94'399","reported_seq":473,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-22T03:15:45.585789+0000","last_change":"2024-09-22T03:15:33.948202+0000","last_active":"2024-09-22T03:15:45.585789+0000","last_peered":"2024-09-22T03:15:45.585789+0000","last_clean":"2024-09-22T03:15:45.585789+0000","last_became_active":"2024-09-22T03:15:33.943275+0000","last_became_peered":"2024-09-22T03:15:33.943275+0000","last_unstale":"2024-09-22T03:15:45.585789+0000","last_undegraded":"2024-09-22T03:15:45.585789+0000","last_fullsized":"2024-09-22T03:15:45.585789+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_clean_scrub_stamp":"2024-09-22T03:15:18.897706+0000","objects_scrubbed":0,"log_size":399,"log_dups_size":0,"ondisk_log_size":399,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3276800,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":400,"num_write_kb":3200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":1179648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.8","version":"95'472","reported_seq":561,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-22T03:15:48.931253+0000","last_change":"2024-09-22T03:15:33.946927+0000","last_active":"2024-09-22T03:15:48.931253+0000","last_peered":"2024-09-22T03:15:48.931253+0000","last_clean":"2024-09-22T03:15:48.931253+0000","last_became_active":"2024-09-22T03:15:33.940267+0000","last_became_peered":"2024-09-22T03:15:33.940267+0000","last_unstale":"2024-09-22T03:15:48.931253+0000","last_undegraded":"2024-09-22T03:15:48.931253+0000","last_fullsized":"2024-09-22T03:15:48.931253+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_clean_scrub_stamp":"2024-09-22T03:15:18.897706+0000","objects_scrubbed":0,"log_size":472,"log_dups_size":0,"ondisk_log_size":472,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3866624,"num_objects":59,"num_object_clones":0,"num_object_copies":177,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":472,"num_write_kb":3776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":1507328,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.6","version":"95'560","reported_seq":622,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-22T03:15:48.922886+0000","last_change":"2024-09-22T03:15:32.999617+0000","last_active":"2024-09-22T03:15:48.922886+0000","last_peered":"2024-09-22T03:15:48.922886+0000","last_clean":"2024-09-22T03:15:48.922886+0000","last_became_active":"2024-09-22T03:15:32.993508+0000","last_became_peered":"2024-09-22T03:15:32.993508+0000","last_unstale":"2024-09-22T03:15:48.922886+0000","last_undegraded":"2024-09-22T03:15:48.922886+0000","last_fullsized":"2024-09-22T03:15:48.922886+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_clean_scrub_stamp":"2024-09-22T03:15:18.897706+0000","objects_scrubbed":0,"log_size":560,"log_dups_size":0,"ondisk_log_size":560,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4587520,"num_objects":70,"num_object_clones":0,"num_object_copies":210,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":560,"num_write_kb":4480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":491520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":124,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-22T03:15:48.924682+0000","last_change":"2024-09-22T03:15:30.926185+0000","last_active":"2024-09-22T03:15:48.924682+0000","last_peered":"2024-09-22T03:15:48.924682+0000","last_clean":"2024-09-22T03:15:48.924682+0000","last_became_active":"2024-09-22T03:15:29.957045+0000","last_became_peered":"2024-09-22T03:15:29.957045+0000","last_unstale":"2024-09-22T03:15:48.924682+0000","last_undegraded":"2024-09-22T03:15:48.924682+0000","last_fullsized":"2024-09-22T03:15:48.924682+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.7","version":"96'639","reported_seq":739,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-22T03:15:49.099772+0000","last_change":"2024-09-22T03:15:33.949109+0000","last_active":"2024-09-22T03:15:49.099772+0000","last_peered":"2024-09-22T03:15:49.099772+0000","last_clean":"2024-09-22T03:15:49.099772+0000","last_became_active":"2024-09-22T03:15:33.943481+0000","last_became_peered":"2024-09-22T03:15:33.943481+0000","last_unstale":"2024-09-22T03:15:49.099772+0000","last_undegraded":"2024-09-22T03:15:49.099772+0000","last_fullsized":"2024-09-22T03:15:49.099772+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_clean_scrub_stamp":"2024-09-22T03:15:18.897706+0000","objects_scrubbed":0,"log_size":639,"log_dups_size":0,"ondisk_log_size":639,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5242880,"num_objects":80,"num_object_clones":0,"num_object_copies":240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":640,"num_write_kb":5120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":1835008,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":201,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-22T03:15:38.911987+0000","last_change":"2024-09-22T03:14:38.225116+0000","last_active":"2024-09-22T03:15:38.911987+0000","last_peered":"2024-09-22T03:15:38.911987+0000","last_clean":"2024-09-22T03:15:38.911987+0000","last_became_active":"2024-09-22T03:14:37.266115+0000","last_became_peered":"2024-09-22T03:14:37.266115+0000","last_unstale":"2024-09-22T03:15:38.911987+0000","last_undegraded":"2024-09-22T03:15:38.911987+0000","last_fullsized":"2024-09-22T03:15:38.911987+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.4","version":"94'455","reported_seq":535,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-22T03:15:45.904412+0000","last_change":"2024-09-22T03:15:33.941381+0000","last_active":"2024-09-22T03:15:45.904412+0000","last_peered":"2024-09-22T03:15:45.904412+0000","last_clean":"2024-09-22T03:15:45.904412+0000","last_became_active":"2024-09-22T03:15:33.938030+0000","last_became_peered":"2024-09-22T03:15:33.938030+0000","last_unstale":"2024-09-22T03:15:45.904412+0000","last_undegraded":"2024-09-22T03:15:45.904412+0000","last_fullsized":"2024-09-22T03:15:45.904412+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_clean_scrub_stamp":"2024-09-22T03:15:18.897706+0000","objects_scrubbed":0,"log_size":455,"log_dups_size":0,"ondisk_log_size":455,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3735552,"num_objects":57,"num_object_clones":0,"num_object_copies":171,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":456,"num_write_kb":3648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1376256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":230,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-22T03:15:38.909203+0000","last_change":"2024-09-22T03:15:30.950741+0000","last_active":"2024-09-22T03:15:38.909203+0000","last_peered":"2024-09-22T03:15:38.909203+0000","last_clean":"2024-09-22T03:15:38.909203+0000","last_became_active":"2024-09-22T03:15:29.979306+0000","last_became_peered":"2024-09-22T03:15:29.979306+0000","last_unstale":"2024-09-22T03:15:38.909203+0000","last_undegraded":"2024-09-22T03:15:38.909203+0000","last_fullsized":"2024-09-22T03:15:38.909203+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.5","version":"96'538","reported_seq":643,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-22T03:15:49.190464+0000","last_change":"2024-09-22T03:15:32.978797+0000","last_active":"2024-09-22T03:15:49.190464+0000","last_peered":"2024-09-22T03:15:49.190464+0000","last_clean":"2024-09-22T03:15:49.190464+0000","last_became_active":"2024-09-22T03:15:32.974824+0000","last_became_peered":"2024-09-22T03:15:32.974824+0000","last_unstale":"2024-09-22T03:15:49.190464+0000","last_undegraded":"2024-09-22T03:15:49.190464+0000","last_fullsized":"2024-09-22T03:15:49.190464+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_clean_scrub_stamp":"2024-09-22T03:15:18.897706+0000","objects_scrubbed":0,"log_size":538,"log_dups_size":0,"ondisk_log_size":538,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4415488,"num_objects":68,"num_object_clones":0,"num_object_copies":204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":539,"num_write_kb":4312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":1966080,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":185,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-22T03:15:48.921686+0000","last_change":"2024-09-22T03:15:30.924363+0000","last_active":"2024-09-22T03:15:48.921686+0000","last_peered":"2024-09-22T03:15:48.921686+0000","last_clean":"2024-09-22T03:15:48.921686+0000","last_became_active":"2024-09-22T03:15:29.958549+0000","last_became_peered":"2024-09-22T03:15:29.958549+0000","last_unstale":"2024-09-22T03:15:48.921686+0000","last_undegraded":"2024-09-22T03:15:48.921686+0000","last_fullsized":"2024-09-22T03:15:48.921686+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"12'97","reported_seq":324,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-22T03:15:38.915351+0000","last_change":"2024-09-22T03:14:37.276946+0000","last_active":"2024-09-22T03:15:38.915351+0000","last_peered":"2024-09-22T03:15:38.915351+0000","last_clean":"2024-09-22T03:15:38.915351+0000","last_became_active":"2024-09-22T03:14:37.273858+0000","last_became_peered":"2024-09-22T03:14:37.273858+0000","last_unstale":"2024-09-22T03:15:38.915351+0000","last_undegraded":"2024-09-22T03:15:38.915351+0000","last_fullsized":"2024-09-22T03:15:38.915351+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.2","version":"95'560","reported_seq":617,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-22T03:15:48.919500+0000","last_change":"2024-09-22T03:15:33.953595+0000","last_active":"2024-09-22T03:15:48.919500+0000","last_peered":"2024-09-22T03:15:48.919500+0000","last_clean":"2024-09-22T03:15:48.919500+0000","last_became_active":"2024-09-22T03:15:33.947435+0000","last_became_peered":"2024-09-22T03:15:33.947435+0000","last_unstale":"2024-09-22T03:15:48.919500+0000","last_undegraded":"2024-09-22T03:15:48.919500+0000","last_fullsized":"2024-09-22T03:15:48.919500+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_clean_scrub_stamp":"2024-09-22T03:15:18.897706+0000","objects_scrubbed":0,"log_size":560,"log_dups_size":0,"ondisk_log_size":560,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4587520,"num_objects":70,"num_object_clones":0,"num_object_copies":210,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":560,"num_write_kb":4480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1376256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":212,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-22T03:15:48.925906+0000","last_change":"2024-09-22T03:14:38.265993+0000","last_active":"2024-09-22T03:15:48.925906+0000","last_peered":"2024-09-22T03:15:48.925906+0000","last_clean":"2024-09-22T03:15:48.925906+0000","last_became_active":"2024-09-22T03:14:37.276817+0000","last_became_peered":"2024-09-22T03:14:37.276817+0000","last_unstale":"2024-09-22T03:15:48.925906+0000","last_undegraded":"2024-09-22T03:15:48.925906+0000","last_fullsized":"2024-09-22T03:15:48.925906+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.3","version":"96'475","reported_seq":529,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-22T03:15:49.097411+0000","last_change":"2024-09-22T03:15:33.961634+0000","last_active":"2024-09-22T03:15:49.097411+0000","last_peered":"2024-09-22T03:15:49.097411+0000","last_clean":"2024-09-22T03:15:49.097411+0000","last_became_active":"2024-09-22T03:15:33.957171+0000","last_became_peered":"2024-09-22T03:15:33.957171+0000","last_unstale":"2024-09-22T03:15:49.097411+0000","last_undegraded":"2024-09-22T03:15:49.097411+0000","last_fullsized":"2024-09-22T03:15:49.097411+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_clean_scrub_stamp":"2024-09-22T03:15:18.897706+0000","objects_scrubbed":0,"log_size":475,"log_dups_size":0,"ondisk_log_size":475,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3899392,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":476,"num_write_kb":3808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1376256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":209,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-22T03:15:38.908795+0000","last_change":"2024-09-22T03:15:30.950892+0000","last_active":"2024-09-22T03:15:38.908795+0000","last_peered":"2024-09-22T03:15:38.908795+0000","last_clean":"2024-09-22T03:15:38.908795+0000","last_became_active":"2024-09-22T03:15:29.963452+0000","last_became_peered":"2024-09-22T03:15:29.963452+0000","last_unstale":"2024-09-22T03:15:38.908795+0000","last_undegraded":"2024-09-22T03:15:38.908795+0000","last_fullsized":"2024-09-22T03:15:38.908795+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.0","version":"94'391","reported_seq":474,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-22T03:15:45.637023+0000","last_change":"2024-09-22T03:15:32.976320+0000","last_active":"2024-09-22T03:15:45.637023+0000","last_peered":"2024-09-22T03:15:45.637023+0000","last_clean":"2024-09-22T03:15:45.637023+0000","last_became_active":"2024-09-22T03:15:32.973547+0000","last_became_peered":"2024-09-22T03:15:32.973547+0000","last_unstale":"2024-09-22T03:15:45.637023+0000","last_undegraded":"2024-09-22T03:15:45.637023+0000","last_fullsized":"2024-09-22T03:15:45.637023+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_clean_scrub_stamp":"2024-09-22T03:15:18.897706+0000","objects_scrubbed":0,"log_size":391,"log_dups_size":0,"ondisk_log_size":391,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3211264,"num_objects":49,"num_object_clones":0,"num_object_copies":147,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":392,"num_write_kb":3136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1327104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":226,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-22T03:15:48.926975+0000","last_change":"2024-09-22T03:15:30.958044+0000","last_active":"2024-09-22T03:15:48.926975+0000","last_peered":"2024-09-22T03:15:48.926975+0000","last_clean":"2024-09-22T03:15:48.926975+0000","last_became_active":"2024-09-22T03:15:29.976915+0000","last_became_peered":"2024-09-22T03:15:29.976915+0000","last_unstale":"2024-09-22T03:15:48.926975+0000","last_undegraded":"2024-09-22T03:15:48.926975+0000","last_fullsized":"2024-09-22T03:15:48.926975+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1","version":"94'343","reported_seq":409,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-22T03:15:46.859362+0000","last_change":"2024-09-22T03:15:33.950769+0000","last_active":"2024-09-22T03:15:46.859362+0000","last_peered":"2024-09-22T03:15:46.859362+0000","last_clean":"2024-09-22T03:15:46.859362+0000","last_became_active":"2024-09-22T03:15:33.945111+0000","last_became_peered":"2024-09-22T03:15:33.945111+0000","last_unstale":"2024-09-22T03:15:46.859362+0000","last_undegraded":"2024-09-22T03:15:46.859362+0000","last_fullsized":"2024-09-22T03:15:46.859362+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_clean_scrub_stamp":"2024-09-22T03:15:18.897706+0000","objects_scrubbed":0,"log_size":343,"log_dups_size":0,"ondisk_log_size":343,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2818048,"num_objects":43,"num_object_clones":0,"num_object_copies":129,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":344,"num_write_kb":2752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":917504,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":221,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-22T03:15:48.927501+0000","last_change":"2024-09-22T03:15:30.958189+0000","last_active":"2024-09-22T03:15:48.927501+0000","last_peered":"2024-09-22T03:15:48.927501+0000","last_clean":"2024-09-22T03:15:48.927501+0000","last_became_active":"2024-09-22T03:15:29.980027+0000","last_became_peered":"2024-09-22T03:15:29.980027+0000","last_unstale":"2024-09-22T03:15:48.927501+0000","last_undegraded":"2024-09-22T03:15:48.927501+0000","last_fullsized":"2024-09-22T03:15:48.927501+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":64151552,"num_objects":980,"num_object_clones":0,"num_object_copies":2940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7831,"num_write_kb":62648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":322,"num_bytes_recovered":21020672,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":86618112,"data_stored":192921600,"data_compressed":9871891,"data_compressed_allocated":86618112,"data_compressed_original":192921600,"omap_allocated":0,"internal_metadata":0},"log_size":7821,"ondisk_log_size":7821,"up":48,"acting":48,"num_store_stats":4},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1180736,"data_compressed":59924,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":87,"seq":373662154772,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":59256,"kb_used_data":21800,"kb_used_omap":0,"kb_used_meta":37440,"kb_avail":94312584,"statfs":{"total":96636764160,"available":96576086016,"internally_reserved":0,"allocated":22323200,"data_stored":62998346,"data_compressed":3149072,"data_compressed_allocated":21512192,"data_compressed_original":62676992,"omap_allocated":0,"internal_metadata":38338560},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":71,"seq":304942678101,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":44244,"kb_used_data":16452,"kb_used_omap":0,"kb_used_meta":27776,"kb_avail":94327596,"statfs":{"total":96636764160,"available":96591458304,"internally_reserved":0,"allocated":16846848,"data_stored":32886744,"data_compressed":1680716,"data_compressed_allocated":16031744,"data_compressed_original":32555008,"omap_allocated":0,"internal_metadata":28442624},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":53,"seq":227633266882,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":68208,"kb_used_data":27680,"kb_used_omap":0,"kb_used_meta":40512,"kb_avail":94303632,"statfs":{"total":96636764160,"available":96566919168,"internally_reserved":0,"allocated":28344320,"data_stored":55863658,"data_compressed":2866543,"data_compressed_allocated":27512832,"data_compressed_original":55525376,"omap_allocated":0,"internal_metadata":41484288},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":73,"seq":313532612678,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":49204,"kb_used_data":21220,"kb_used_omap":0,"kb_used_meta":27968,"kb_avail":94322636,"statfs":{"total":96636764160,"available":96586379264,"internally_reserved":0,"allocated":21729280,"data_stored":42684357,"data_compressed":2186345,"data_compressed_allocated":20918272,"data_compressed_original":42352640,"omap_allocated":0,"internal_metadata":28639232},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29962,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29962,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":21311488,"data_stored":42647552,"data_compressed":2201221,"data_compressed_allocated":21311488,"data_compressed_original":42647552,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":27643904,"data_stored":55296000,"data_compressed":2854347,"data_compressed_allocated":27643904,"data_compressed_original":55296000,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":15990784,"data_stored":31981568,"data_compressed":1650754,"data_compressed_allocated":15990784,"data_compressed_original":31981568,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":21671936,"data_stored":62996480,"data_compressed":3165569,"data_compressed_allocated":21671936,"data_compressed_original":62996480,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-22T03:15:51.835 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-09-22T03:15:51.836 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-22T03:15:51.836 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-09-22T03:15:51.836 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-09-22T03:15:51.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:15:51.961+0000 7fe65d46c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:15:51.974 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:15:51.972+0000 7fe65d46c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:15:51.982 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 31 16 9086 9070 2.2856 2.35156 0.060283 0.0546364 2024-09-22T03:15:51.991 INFO:teuthology.orchestra.run.smithi153.stderr:no valid command found; 10 closest matches: 2024-09-22T03:15:51.991 INFO:teuthology.orchestra.run.smithi153.stderr:config get 2024-09-22T03:15:51.991 INFO:teuthology.orchestra.run.smithi153.stderr:config help 2024-09-22T03:15:51.991 INFO:teuthology.orchestra.run.smithi153.stderr:config set ... 2024-09-22T03:15:51.991 INFO:teuthology.orchestra.run.smithi153.stderr:config show 2024-09-22T03:15:51.992 INFO:teuthology.orchestra.run.smithi153.stderr:deep_scrub 2024-09-22T03:15:51.992 INFO:teuthology.orchestra.run.smithi153.stderr:dump_historic_ops 2024-09-22T03:15:51.992 INFO:teuthology.orchestra.run.smithi153.stderr:dump_historic_slow_ops 2024-09-22T03:15:51.992 INFO:teuthology.orchestra.run.smithi153.stderr:dump_metrics [] 2024-09-22T03:15:51.992 INFO:teuthology.orchestra.run.smithi153.stderr:dump_ops_in_flight 2024-09-22T03:15:51.992 INFO:teuthology.orchestra.run.smithi153.stderr:dump_pgstate_history 2024-09-22T03:15:51.992 INFO:teuthology.orchestra.run.smithi153.stderr:Error EINVAL: invalid command 2024-09-22T03:15:51.995 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-22T03:15:51.995 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-22T03:15:51.996 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-09-22T03:15:52.137 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:15:52.127+0000 7f10624c3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:15:52.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:15:52.138+0000 7f10624c3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:15:52.169 INFO:teuthology.orchestra.run.smithi153.stderr:no valid command found; 10 closest matches: 2024-09-22T03:15:52.169 INFO:teuthology.orchestra.run.smithi153.stderr:config get 2024-09-22T03:15:52.169 INFO:teuthology.orchestra.run.smithi153.stderr:config help 2024-09-22T03:15:52.169 INFO:teuthology.orchestra.run.smithi153.stderr:config set ... 2024-09-22T03:15:52.169 INFO:teuthology.orchestra.run.smithi153.stderr:config show 2024-09-22T03:15:52.169 INFO:teuthology.orchestra.run.smithi153.stderr:deep_scrub 2024-09-22T03:15:52.169 INFO:teuthology.orchestra.run.smithi153.stderr:dump_historic_ops 2024-09-22T03:15:52.169 INFO:teuthology.orchestra.run.smithi153.stderr:dump_historic_slow_ops 2024-09-22T03:15:52.169 INFO:teuthology.orchestra.run.smithi153.stderr:dump_metrics [] 2024-09-22T03:15:52.169 INFO:teuthology.orchestra.run.smithi153.stderr:dump_ops_in_flight 2024-09-22T03:15:52.169 INFO:teuthology.orchestra.run.smithi153.stderr:dump_pgstate_history 2024-09-22T03:15:52.169 INFO:teuthology.orchestra.run.smithi153.stderr:Error EINVAL: invalid command 2024-09-22T03:15:52.172 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-22T03:15:52.172 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-22T03:15:52.172 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-09-22T03:15:52.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:15:52.284+0000 7f9e04e6c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:15:52.296 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:15:52.293+0000 7f9e04e6c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:15:52.314 INFO:teuthology.orchestra.run.smithi153.stderr:no valid command found; 10 closest matches: 2024-09-22T03:15:52.314 INFO:teuthology.orchestra.run.smithi153.stderr:config get 2024-09-22T03:15:52.314 INFO:teuthology.orchestra.run.smithi153.stderr:config help 2024-09-22T03:15:52.314 INFO:teuthology.orchestra.run.smithi153.stderr:config set ... 2024-09-22T03:15:52.314 INFO:teuthology.orchestra.run.smithi153.stderr:config show 2024-09-22T03:15:52.314 INFO:teuthology.orchestra.run.smithi153.stderr:deep_scrub 2024-09-22T03:15:52.314 INFO:teuthology.orchestra.run.smithi153.stderr:dump_historic_ops 2024-09-22T03:15:52.314 INFO:teuthology.orchestra.run.smithi153.stderr:dump_historic_slow_ops 2024-09-22T03:15:52.314 INFO:teuthology.orchestra.run.smithi153.stderr:dump_metrics [] 2024-09-22T03:15:52.315 INFO:teuthology.orchestra.run.smithi153.stderr:dump_ops_in_flight 2024-09-22T03:15:52.315 INFO:teuthology.orchestra.run.smithi153.stderr:dump_pgstate_history 2024-09-22T03:15:52.315 INFO:teuthology.orchestra.run.smithi153.stderr:Error EINVAL: invalid command 2024-09-22T03:15:52.317 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-22T03:15:52.317 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-22T03:15:52.317 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-09-22T03:15:52.459 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:15:52.450+0000 7f0d3a157640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:15:52.465 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:15:52.462+0000 7f0d3a157640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:15:52.487 INFO:teuthology.orchestra.run.smithi153.stderr:no valid command found; 10 closest matches: 2024-09-22T03:15:52.487 INFO:teuthology.orchestra.run.smithi153.stderr:config get 2024-09-22T03:15:52.487 INFO:teuthology.orchestra.run.smithi153.stderr:config help 2024-09-22T03:15:52.487 INFO:teuthology.orchestra.run.smithi153.stderr:config set ... 2024-09-22T03:15:52.487 INFO:teuthology.orchestra.run.smithi153.stderr:config show 2024-09-22T03:15:52.487 INFO:teuthology.orchestra.run.smithi153.stderr:deep_scrub 2024-09-22T03:15:52.487 INFO:teuthology.orchestra.run.smithi153.stderr:dump_historic_ops 2024-09-22T03:15:52.487 INFO:teuthology.orchestra.run.smithi153.stderr:dump_historic_slow_ops 2024-09-22T03:15:52.488 INFO:teuthology.orchestra.run.smithi153.stderr:dump_metrics [] 2024-09-22T03:15:52.488 INFO:teuthology.orchestra.run.smithi153.stderr:dump_ops_in_flight 2024-09-22T03:15:52.488 INFO:teuthology.orchestra.run.smithi153.stderr:dump_pgstate_history 2024-09-22T03:15:52.488 INFO:teuthology.orchestra.run.smithi153.stderr:Error EINVAL: invalid command 2024-09-22T03:15:52.490 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-22T03:15:52.491 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-22T03:15:52.651 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-22T03:15:52.802 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:15:52.799+0000 7f906b74e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:15:52.807 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:15:52.805+0000 7f906b74e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:15:52.982 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 32 16 9395 9379 2.28961 2.41406 0.0848928 0.0544985 2024-09-22T03:15:53.982 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 33 15 9723 9708 2.2981 2.57031 0.0656567 0.0543478 2024-09-22T03:15:54.942 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-22T03:15:54.982 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 34 16 10047 10031 2.30473 2.52344 0.0431302 0.0541939 2024-09-22T03:15:55.074 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:15:55.070+0000 7f8dbb357640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:15:55.078 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:15:55.076+0000 7f8dbb357640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:15:55.982 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 35 16 10363 10347 2.30941 2.46875 0.0362244 0.0540903 2024-09-22T03:15:56.982 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 36 16 10706 10690 2.31969 2.67969 0.0287674 0.0538422 2024-09-22T03:15:57.252 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-22T03:15:57.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:15:57.376+0000 7f4d39b57640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:15:57.384 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:15:57.383+0000 7f4d39b57640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:15:57.491 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [1, 2, 0, 3] 2024-09-22T03:15:57.492 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-22T03:15:57.492 INFO:tasks.thrashosds.thrasher:Killing osd 2, live_osds are [1, 2, 0, 3] 2024-09-22T03:15:57.492 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-09-22T03:15:57.492 INFO:teuthology.orchestra.run:waiting for 300 2024-09-22T03:15:57.982 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 37 16 10864 10848 2.29035 1.23438 0.0361056 0.0538527 2024-09-22T03:15:58.983 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 38 16 10864 10848 2.23008 0 - 0.0538527 2024-09-22T03:15:59.524 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-22T03:15:59.638 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:15:59.635+0000 7fa79c26c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:15:59.641 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:15:59.640+0000 7fa79c26c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:15:59.983 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 39 16 10864 10848 2.17289 0 - 0.0538527 2024-09-22T03:16:00.983 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:2024-09-22T03:16:00.982549+0000 min lat: 0.00692898 max lat: 2.41342 avg lat: 0.0538527 2024-09-22T03:16:00.983 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-22T03:16:00.983 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 40 16 10864 10848 2.11857 0 - 0.0538527 2024-09-22T03:16:01.983 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 41 16 10864 10848 2.06689 0 - 0.0538527 2024-09-22T03:16:02.983 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 42 16 10864 10848 2.01768 0 - 0.0538527 2024-09-22T03:16:03.051 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~0s 2024-09-22T03:16:03.593 INFO:tasks.ceph.osd.2:Stopped 2024-09-22T03:16:03.983 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 43 16 10864 10848 1.97075 0 - 0.0538527 2024-09-22T03:16:04.983 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 44 16 10864 10848 1.92596 0 - 0.0538527 2024-09-22T03:16:05.983 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 45 16 10864 10848 1.88316 0 - 0.0538527 2024-09-22T03:16:06.984 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 46 16 10864 10848 1.84222 0 - 0.0538527 2024-09-22T03:16:07.984 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 47 16 10864 10848 1.80303 0 - 0.0538527 2024-09-22T03:16:08.554 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 has been restored 2024-09-22T03:16:08.593 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [2] live_osds: [1, 0, 3] 2024-09-22T03:16:08.593 INFO:tasks.thrashosds.thrasher:Reviving osd 2 2024-09-22T03:16:08.593 INFO:tasks.ceph.osd.2:Restarting daemon 2024-09-22T03:16:08.594 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2>> /var/log/ceph/ceph-osd.2.log 2024-09-22T03:16:08.596 INFO:tasks.ceph.osd.2:Started 2024-09-22T03:16:08.597 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-22T03:16:08.707 INFO:teuthology.orchestra.run.smithi196.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-09-22T03:16:08.710 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-22T03:16:08.710 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2024-09-22T03:16:08.984 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 48 16 10864 10848 1.76546 0 - 0.0538527 2024-09-22T03:16:09.984 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 49 16 10864 10848 1.72943 0 - 0.0538527 2024-09-22T03:16:10.984 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 50 16 10864 10848 1.69484 0 - 0.0538527 2024-09-22T03:16:11.984 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 51 16 10864 10848 1.66161 0 - 0.0538527 2024-09-22T03:16:12.984 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 52 16 10864 10848 1.62965 0 - 0.0538527 2024-09-22T03:16:13.711 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-22T03:16:13.807 INFO:teuthology.orchestra.run.smithi196.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-09-22T03:16:13.809 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-22T03:16:13.810 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2024-09-22T03:16:13.985 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 53 16 10864 10848 1.5989 0 - 0.0538527 2024-09-22T03:16:14.985 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 54 16 10864 10848 1.56929 0 - 0.0538527 2024-09-22T03:16:15.926 INFO:teuthology.orchestra.run.smithi153.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-09-22T03:16:15.985 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 55 16 10864 10848 1.54076 0 - 0.0538527 2024-09-22T03:16:15.993 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-22T03:16:15.994 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-22T03:16:16.985 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 56 16 10864 10848 1.51324 0 - 0.0538527 2024-09-22T03:16:17.985 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 57 16 10864 10848 1.4867 0 - 0.0538527 2024-09-22T03:16:17.994 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-22T03:16:18.133 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:16:18.129+0000 7f06748a9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:16:18.137 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:16:18.135+0000 7f06748a9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:16:18.811 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-22T03:16:18.986 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-09-22T03:16:18.986 INFO:teuthology.orchestra.run.smithi196.stdout: "ops_in_flight": [ 2024-09-22T03:16:18.986 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-22T03:16:18.986 INFO:teuthology.orchestra.run.smithi196.stdout: "type": "client_request", 2024-09-22T03:16:18.986 INFO:teuthology.orchestra.run.smithi196.stdout: "id": 269, 2024-09-22T03:16:18.986 INFO:teuthology.orchestra.run.smithi196.stdout: "detail": { 2024-09-22T03:16:18.986 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:18.986 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:16:18.986 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.514556+0000" 2024-09-22T03:16:18.987 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:18.987 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:18.987 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:16:18.987 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.514565+0000" 2024-09-22T03:16:18.987 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:18.987 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:18.987 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:18.987 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:18.987 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:18.987 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:18.987 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:16:18.987 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.515089+0000" 2024-09-22T03:16:18.987 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:18.987 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:18.987 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:18.988 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:18.988 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:18.988 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:18.988 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:16:18.988 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:18.988 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:18.988 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:18.988 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:18.988 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:18.988 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:18.988 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:18.988 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:16:18.988 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.517717+0000" 2024-09-22T03:16:18.989 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:18.989 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:18.989 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:18.989 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:18.989 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:18.989 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:18.989 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:16:18.989 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.518155+0000" 2024-09-22T03:16:18.989 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:18.989 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:18.989 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:18.989 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:18.989 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:18.990 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:18.990 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:16:18.990 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:18.990 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:18.990 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:18.990 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:18.990 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:18.990 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:18.990 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:18.990 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:16:18.990 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.558673+0000" 2024-09-22T03:16:18.990 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:18.990 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:18.990 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:16:18.991 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.518568+0000" 2024-09-22T03:16:18.991 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:18.991 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:18.991 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:18.991 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:18.991 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:18.991 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:18.991 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:16:18.991 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:18.991 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:18.991 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:18.991 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:18.991 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:18.991 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:18.991 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:18.992 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:16:18.992 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.519250+0000" 2024-09-22T03:16:18.992 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:18.992 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:18.992 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:18.992 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:18.992 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:18.992 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:18.992 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:16:18.992 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:18.992 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:18.992 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:18.993 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:18.993 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:18.993 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:18.993 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:18.993 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:16:18.993 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.519640+0000" 2024-09-22T03:16:18.993 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:18.993 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:18.993 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:18.993 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:18.993 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:18.993 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:18.993 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:16:18.993 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:18.994 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:18.994 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:18.994 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:18.994 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:18.994 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:18.994 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:18.994 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:16:18.994 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:18.994 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:18.994 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:18.994 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:18.994 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:18.994 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:18.994 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:18.995 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:16:18.995 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:18.995 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:18.995 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:18.995 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:18.995 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:18.995 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:18.995 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:18.995 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:16:18.995 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.521064+0000" 2024-09-22T03:16:18.995 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:18.995 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:18.995 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:18.995 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:18.996 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:18.996 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:18.996 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:16:18.996 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:18.996 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:18.996 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:18.996 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:18.996 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:18.996 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:18.996 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:18.996 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:16:18.996 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.521812+0000" 2024-09-22T03:16:18.996 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:18.996 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:18.997 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:18.997 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:18.997 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:18.997 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:18.997 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:16:18.997 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:18.997 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:18.997 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:18.997 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:18.997 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:18.997 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:18.997 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:18.997 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:16:18.997 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.557646+0000" 2024-09-22T03:16:18.998 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:18.998 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:18.998 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:18.998 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:18.998 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:18.998 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:18.998 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:16:18.998 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:18.998 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:16:18.998 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:16:18.998 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:18.998 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-22T03:16:18.998 INFO:teuthology.orchestra.run.smithi196.stdout: "type": "client_request", 2024-09-22T03:16:18.998 INFO:teuthology.orchestra.run.smithi196.stdout: "id": 270, 2024-09-22T03:16:18.999 INFO:teuthology.orchestra.run.smithi196.stdout: "detail": { 2024-09-22T03:16:18.999 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:18.999 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:16:18.999 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.522640+0000" 2024-09-22T03:16:18.999 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:18.999 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:18.999 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:16:18.999 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.522649+0000" 2024-09-22T03:16:19.000 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.000 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.000 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.000 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.000 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.000 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.000 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:16:19.001 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.523158+0000" 2024-09-22T03:16:19.001 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.001 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.001 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.001 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.001 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.001 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.001 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:16:19.001 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.002 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.002 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.002 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.002 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.002 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.002 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.002 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:16:19.002 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.525427+0000" 2024-09-22T03:16:19.002 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.002 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.002 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.002 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.002 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.003 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.003 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:16:19.003 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.525693+0000" 2024-09-22T03:16:19.003 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.003 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.003 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.003 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.003 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.003 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.003 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:16:19.003 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.003 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.003 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.003 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.004 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.004 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.004 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.004 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:16:19.004 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.564069+0000" 2024-09-22T03:16:19.004 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.004 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.004 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:16:19.004 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.526038+0000" 2024-09-22T03:16:19.004 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.004 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.004 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.004 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.004 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.005 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.005 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:16:19.005 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.005 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.005 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.005 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.005 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.005 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.005 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.005 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:16:19.005 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.526326+0000" 2024-09-22T03:16:19.005 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.005 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.005 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.006 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.006 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.006 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.006 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:16:19.006 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.006 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.006 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.006 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.006 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.006 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.006 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.006 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:16:19.006 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.527152+0000" 2024-09-22T03:16:19.007 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.007 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.007 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.007 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.007 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.007 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.007 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:16:19.007 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.007 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.007 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.007 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.007 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.007 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.007 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.008 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:16:19.008 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.008 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.008 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.008 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.008 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.008 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.008 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.008 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:16:19.008 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.008 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.008 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.008 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.008 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.008 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.009 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.009 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:16:19.009 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.527329+0000" 2024-09-22T03:16:19.009 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.009 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.009 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.009 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.009 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.009 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.009 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:16:19.010 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.010 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.010 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.010 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.010 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.010 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.010 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.011 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:16:19.011 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.528013+0000" 2024-09-22T03:16:19.011 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.011 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.011 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.011 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.012 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.012 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.012 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:16:19.012 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.012 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.012 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.013 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.013 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.013 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.013 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.013 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:16:19.013 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.563122+0000" 2024-09-22T03:16:19.013 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.014 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.014 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.014 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.014 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.014 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.014 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:16:19.014 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.015 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:16:19.015 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:16:19.015 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.015 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-22T03:16:19.015 INFO:teuthology.orchestra.run.smithi196.stdout: "type": "client_request", 2024-09-22T03:16:19.015 INFO:teuthology.orchestra.run.smithi196.stdout: "id": 271, 2024-09-22T03:16:19.015 INFO:teuthology.orchestra.run.smithi196.stdout: "detail": { 2024-09-22T03:16:19.015 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.015 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:16:19.016 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.528284+0000" 2024-09-22T03:16:19.016 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.016 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.016 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:16:19.016 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.528288+0000" 2024-09-22T03:16:19.016 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.016 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.017 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.017 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.017 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.017 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.017 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:16:19.017 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.528645+0000" 2024-09-22T03:16:19.017 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.017 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.018 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.018 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.018 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.018 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.018 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:16:19.018 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.018 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.018 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.019 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.019 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.019 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.019 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.019 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:16:19.019 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.530166+0000" 2024-09-22T03:16:19.019 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.020 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.020 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.020 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.020 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.020 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.020 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:16:19.020 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.530274+0000" 2024-09-22T03:16:19.020 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.021 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.021 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.021 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.021 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.021 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.021 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:16:19.021 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.021 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.021 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.021 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.021 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.021 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.021 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.022 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:16:19.022 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.569495+0000" 2024-09-22T03:16:19.022 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.022 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.022 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:16:19.022 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.531344+0000" 2024-09-22T03:16:19.022 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.022 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.022 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.022 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.022 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.023 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.023 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:16:19.023 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.023 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.023 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.023 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.023 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.023 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.023 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.023 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:16:19.023 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.532106+0000" 2024-09-22T03:16:19.023 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.023 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.023 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.023 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.024 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.025 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.025 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:16:19.025 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.025 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.025 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.025 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.026 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.026 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.026 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.026 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:16:19.026 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.532754+0000" 2024-09-22T03:16:19.026 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.026 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.026 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.026 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.026 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.026 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.026 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:16:19.026 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.027 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.027 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.027 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.027 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.027 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.027 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.027 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:16:19.027 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.027 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.027 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.027 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.027 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.027 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.027 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.028 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:16:19.028 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.028 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.028 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.028 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.028 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.028 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.028 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.028 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:16:19.028 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.532992+0000" 2024-09-22T03:16:19.028 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.028 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.029 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.029 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.029 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.029 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.029 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:16:19.029 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.029 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.029 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.029 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.029 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.029 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.029 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.029 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:16:19.030 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.534211+0000" 2024-09-22T03:16:19.030 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.030 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.030 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.030 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.030 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.030 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.030 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:16:19.030 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.030 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.030 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.030 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.030 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.030 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.031 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.031 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:16:19.031 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.568210+0000" 2024-09-22T03:16:19.031 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.031 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.031 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.031 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.031 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.031 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.031 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:16:19.031 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.031 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:16:19.031 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:16:19.032 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.032 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-22T03:16:19.032 INFO:teuthology.orchestra.run.smithi196.stdout: "type": "client_request", 2024-09-22T03:16:19.032 INFO:teuthology.orchestra.run.smithi196.stdout: "id": 308, 2024-09-22T03:16:19.032 INFO:teuthology.orchestra.run.smithi196.stdout: "detail": { 2024-09-22T03:16:19.032 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.032 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:16:19.032 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.762670+0000" 2024-09-22T03:16:19.032 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.032 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.032 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:16:19.032 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.762674+0000" 2024-09-22T03:16:19.032 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.032 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.033 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.033 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.033 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.033 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.033 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:16:19.033 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.762754+0000" 2024-09-22T03:16:19.033 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.033 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.033 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.033 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.033 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.033 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.033 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:16:19.033 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.034 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.034 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.034 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.034 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.034 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.034 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.034 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:16:19.034 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.762802+0000" 2024-09-22T03:16:19.034 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.034 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.034 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.034 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.034 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.034 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.035 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:16:19.035 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.762852+0000" 2024-09-22T03:16:19.035 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.035 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.035 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.035 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.035 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.035 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.035 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:16:19.035 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.035 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.035 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.035 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.035 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.036 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.036 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.036 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:16:19.036 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.776212+0000" 2024-09-22T03:16:19.036 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.036 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.036 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:16:19.036 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.763058+0000" 2024-09-22T03:16:19.036 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.036 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.036 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.036 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.036 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.036 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.037 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:16:19.037 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.037 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.037 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.037 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.037 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.037 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.037 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.037 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:16:19.037 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.763161+0000" 2024-09-22T03:16:19.037 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.037 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.038 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.038 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.038 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.038 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.038 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:16:19.038 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.038 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.038 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.038 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.038 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.038 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.038 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.038 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:16:19.038 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.763246+0000" 2024-09-22T03:16:19.039 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.039 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.039 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.039 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.039 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.039 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.039 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:16:19.039 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.039 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.039 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.039 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.039 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.039 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.039 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.040 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:16:19.040 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.040 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.040 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.040 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.040 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.040 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.040 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.040 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:16:19.040 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.040 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.040 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.040 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.040 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.041 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.041 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.041 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:16:19.041 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.763389+0000" 2024-09-22T03:16:19.041 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.041 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.041 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.041 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.041 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.041 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.041 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:16:19.041 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.042 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.042 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.042 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.042 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.042 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.042 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.042 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:16:19.042 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.764007+0000" 2024-09-22T03:16:19.042 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.042 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.042 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.042 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.043 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.043 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.043 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:16:19.043 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.043 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.043 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.043 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.043 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.043 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.043 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.043 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:16:19.043 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.775904+0000" 2024-09-22T03:16:19.043 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.043 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.044 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.044 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.044 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.044 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.044 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:16:19.044 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.044 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:16:19.044 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:16:19.044 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.044 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-22T03:16:19.044 INFO:teuthology.orchestra.run.smithi196.stdout: "type": "client_request", 2024-09-22T03:16:19.044 INFO:teuthology.orchestra.run.smithi196.stdout: "id": 310, 2024-09-22T03:16:19.044 INFO:teuthology.orchestra.run.smithi196.stdout: "detail": { 2024-09-22T03:16:19.044 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.045 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:16:19.045 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.767550+0000" 2024-09-22T03:16:19.045 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.045 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.045 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:16:19.045 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.767559+0000" 2024-09-22T03:16:19.045 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.045 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.045 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.045 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.045 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.045 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.045 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:16:19.045 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.767680+0000" 2024-09-22T03:16:19.046 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.046 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.046 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.046 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.046 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.046 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.046 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:16:19.046 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.046 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.046 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.046 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.046 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.046 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.046 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.047 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:16:19.047 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.767777+0000" 2024-09-22T03:16:19.047 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.047 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.047 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.047 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.047 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.047 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.047 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:16:19.047 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.768318+0000" 2024-09-22T03:16:19.047 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.047 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.047 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.047 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.048 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.048 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.048 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:16:19.048 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.048 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.048 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.048 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.048 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.048 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.048 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.048 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:16:19.048 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.793245+0000" 2024-09-22T03:16:19.048 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.048 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.049 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:16:19.049 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.768586+0000" 2024-09-22T03:16:19.049 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.049 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.049 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.049 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.049 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.049 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.049 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:16:19.049 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.049 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.049 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.049 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.049 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.049 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.050 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.050 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:16:19.050 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.768677+0000" 2024-09-22T03:16:19.050 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.050 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.050 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.050 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.050 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.050 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.050 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:16:19.050 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.050 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.051 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.051 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.051 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.051 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.051 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.051 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:16:19.051 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.768763+0000" 2024-09-22T03:16:19.051 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.051 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.051 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.051 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.051 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.051 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.051 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:16:19.052 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.052 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.052 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.052 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.052 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.052 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.052 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.052 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:16:19.052 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.052 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.052 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.052 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.052 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.053 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.053 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.053 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:16:19.053 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.053 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.053 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.053 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.053 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.053 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.053 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.053 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:16:19.053 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.768886+0000" 2024-09-22T03:16:19.053 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.053 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.053 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.054 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.054 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.054 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.054 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:16:19.054 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.054 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.054 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.054 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.054 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.054 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.054 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.054 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:16:19.055 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.769126+0000" 2024-09-22T03:16:19.055 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.055 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.055 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.055 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.055 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.055 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.055 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:16:19.055 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.055 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.055 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.055 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.055 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.055 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.056 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.056 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:16:19.056 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.790302+0000" 2024-09-22T03:16:19.056 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.056 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.056 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.056 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.056 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.056 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.056 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:16:19.056 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.056 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:16:19.056 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:16:19.056 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.056 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-22T03:16:19.057 INFO:teuthology.orchestra.run.smithi196.stdout: "type": "client_request", 2024-09-22T03:16:19.057 INFO:teuthology.orchestra.run.smithi196.stdout: "id": 312, 2024-09-22T03:16:19.057 INFO:teuthology.orchestra.run.smithi196.stdout: "detail": { 2024-09-22T03:16:19.057 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.057 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:16:19.057 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.777209+0000" 2024-09-22T03:16:19.057 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.057 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.057 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:16:19.057 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.777215+0000" 2024-09-22T03:16:19.057 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.057 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.057 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.057 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.058 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.058 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.058 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:16:19.058 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.777283+0000" 2024-09-22T03:16:19.058 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.058 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.058 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.058 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.058 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.058 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.058 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:16:19.058 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.058 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.058 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.059 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.059 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.059 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.059 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.059 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:16:19.059 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.777497+0000" 2024-09-22T03:16:19.059 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.059 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.059 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.059 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.059 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.059 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.059 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:16:19.059 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.777582+0000" 2024-09-22T03:16:19.059 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.060 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.060 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.060 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.060 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.060 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.060 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:16:19.060 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.060 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.060 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.060 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.060 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.060 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.060 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.061 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:16:19.061 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.805444+0000" 2024-09-22T03:16:19.061 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.061 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.061 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:16:19.061 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.777801+0000" 2024-09-22T03:16:19.061 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.061 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.061 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.061 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.061 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.061 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.061 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:16:19.061 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.062 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.062 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.062 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.062 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.062 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.062 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.062 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:16:19.062 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.777931+0000" 2024-09-22T03:16:19.062 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.062 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.062 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.062 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.062 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.062 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.063 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:16:19.063 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.063 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.063 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.063 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.063 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.063 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.063 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.063 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:16:19.063 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.778058+0000" 2024-09-22T03:16:19.063 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.063 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.063 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.063 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.064 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.064 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.064 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:16:19.064 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.064 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.064 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.064 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.064 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.064 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.064 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.064 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:16:19.064 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.064 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.064 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.065 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.065 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.065 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.065 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.065 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:16:19.065 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.065 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.065 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.065 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.065 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.065 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.065 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.065 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:16:19.065 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.778532+0000" 2024-09-22T03:16:19.065 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.066 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.066 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.066 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.066 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.066 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.066 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:16:19.066 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.066 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.066 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.066 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.066 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.066 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.066 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.066 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:16:19.067 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.778828+0000" 2024-09-22T03:16:19.067 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.067 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.067 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.067 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.067 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.067 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.067 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:16:19.067 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.067 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.067 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.067 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.067 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.067 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.068 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.068 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:16:19.068 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.804851+0000" 2024-09-22T03:16:19.068 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.068 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.068 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.068 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.068 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.068 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.068 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:16:19.068 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.068 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:16:19.068 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:16:19.068 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.069 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-22T03:16:19.069 INFO:teuthology.orchestra.run.smithi196.stdout: "type": "client_request", 2024-09-22T03:16:19.069 INFO:teuthology.orchestra.run.smithi196.stdout: "id": 313, 2024-09-22T03:16:19.069 INFO:teuthology.orchestra.run.smithi196.stdout: "detail": { 2024-09-22T03:16:19.069 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.069 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:16:19.069 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.781684+0000" 2024-09-22T03:16:19.069 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.069 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.069 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:16:19.069 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.781690+0000" 2024-09-22T03:16:19.069 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.069 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.069 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.070 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.070 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.070 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.070 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:16:19.070 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.781787+0000" 2024-09-22T03:16:19.070 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.070 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.070 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.070 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.070 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.070 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.070 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:16:19.070 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.070 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.071 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.071 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.071 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.071 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.071 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.071 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:16:19.071 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.782018+0000" 2024-09-22T03:16:19.071 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.071 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.071 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.071 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.071 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.071 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.071 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:16:19.072 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.782135+0000" 2024-09-22T03:16:19.072 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.072 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.072 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.072 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.072 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.072 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.072 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:16:19.072 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.072 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.072 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.072 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.072 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.072 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.073 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.073 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:16:19.073 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.813495+0000" 2024-09-22T03:16:19.073 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.073 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.073 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:16:19.073 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.782562+0000" 2024-09-22T03:16:19.073 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.073 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.073 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.073 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.073 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.073 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.073 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:16:19.074 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.074 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.074 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.074 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.074 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.074 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.074 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.074 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:16:19.074 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.782724+0000" 2024-09-22T03:16:19.074 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.074 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.074 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.074 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.074 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.075 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.075 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:16:19.075 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.075 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.075 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.075 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.075 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.075 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.075 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.075 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:16:19.075 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.783219+0000" 2024-09-22T03:16:19.075 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.075 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.075 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.076 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.076 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.076 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.076 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:16:19.076 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.076 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.076 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.076 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.076 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.076 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.076 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.076 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:16:19.076 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.076 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.077 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.077 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.077 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.077 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.077 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.077 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:16:19.077 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.077 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.077 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.077 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.077 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.077 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.077 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.077 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:16:19.078 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.783577+0000" 2024-09-22T03:16:19.078 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.078 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.078 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.078 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.078 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.078 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.078 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:16:19.078 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.078 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.078 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.078 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.078 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.078 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.079 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.079 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:16:19.079 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.784520+0000" 2024-09-22T03:16:19.079 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.079 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.079 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.079 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.079 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.079 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.079 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:16:19.079 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.079 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.079 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.079 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.080 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.080 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.080 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.080 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:16:19.080 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.810204+0000" 2024-09-22T03:16:19.080 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.080 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.080 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.080 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.080 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.080 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.080 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:16:19.080 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.080 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:16:19.080 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:16:19.081 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.081 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-22T03:16:19.081 INFO:teuthology.orchestra.run.smithi196.stdout: "type": "client_request", 2024-09-22T03:16:19.081 INFO:teuthology.orchestra.run.smithi196.stdout: "id": 315, 2024-09-22T03:16:19.081 INFO:teuthology.orchestra.run.smithi196.stdout: "detail": { 2024-09-22T03:16:19.081 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.081 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:16:19.081 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.787406+0000" 2024-09-22T03:16:19.081 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.081 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.081 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:16:19.081 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.787415+0000" 2024-09-22T03:16:19.081 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.081 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.082 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.082 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.082 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.082 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.082 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:16:19.082 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.787564+0000" 2024-09-22T03:16:19.082 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.082 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.082 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.082 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.082 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.082 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.082 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:16:19.082 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.083 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.083 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.083 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.083 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.083 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.083 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.083 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:16:19.084 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.788144+0000" 2024-09-22T03:16:19.084 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.084 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.084 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.084 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.084 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.085 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.085 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:16:19.085 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.788296+0000" 2024-09-22T03:16:19.085 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.085 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.085 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.085 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.085 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.086 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.086 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:16:19.086 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.086 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.086 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.086 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.086 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.087 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.087 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.087 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:16:19.087 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.825680+0000" 2024-09-22T03:16:19.087 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.087 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.087 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:16:19.087 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.788561+0000" 2024-09-22T03:16:19.087 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.088 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.088 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.088 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.088 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.088 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.088 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:16:19.088 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.089 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.089 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.089 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.089 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.089 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.089 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.090 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:16:19.090 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.788983+0000" 2024-09-22T03:16:19.090 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.090 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.090 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.090 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.090 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.091 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.091 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:16:19.091 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.091 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.091 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.091 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.091 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.091 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.091 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.091 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:16:19.092 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.789138+0000" 2024-09-22T03:16:19.092 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.092 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.092 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.092 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.092 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.092 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.092 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:16:19.092 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.092 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.092 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.092 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.092 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.092 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.093 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.093 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:16:19.093 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.093 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.093 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.093 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.093 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.093 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.093 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.093 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:16:19.093 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.093 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.093 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.093 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.094 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.094 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.094 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.094 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:16:19.094 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.789334+0000" 2024-09-22T03:16:19.094 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.094 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.094 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.094 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.094 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.094 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.094 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:16:19.094 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.094 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.095 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.095 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.095 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.095 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.095 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.095 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:16:19.095 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.790084+0000" 2024-09-22T03:16:19.095 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.095 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.095 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.095 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.095 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.095 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.095 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:16:19.095 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.096 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.096 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.096 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.096 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.096 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.096 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.096 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:16:19.096 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.824386+0000" 2024-09-22T03:16:19.096 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.096 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.096 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.096 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.096 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.096 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.097 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:16:19.097 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.097 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:16:19.097 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:16:19.097 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.097 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-22T03:16:19.097 INFO:teuthology.orchestra.run.smithi196.stdout: "type": "client_request", 2024-09-22T03:16:19.097 INFO:teuthology.orchestra.run.smithi196.stdout: "id": 316, 2024-09-22T03:16:19.097 INFO:teuthology.orchestra.run.smithi196.stdout: "detail": { 2024-09-22T03:16:19.097 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.097 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:16:19.097 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.794366+0000" 2024-09-22T03:16:19.097 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.097 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.098 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:16:19.098 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.794377+0000" 2024-09-22T03:16:19.098 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.098 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.098 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.098 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.098 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.098 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.098 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:16:19.098 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.794534+0000" 2024-09-22T03:16:19.098 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.098 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.098 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.098 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.099 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.099 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.099 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:16:19.099 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.099 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.099 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.099 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.099 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.099 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.099 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.099 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:16:19.099 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.794649+0000" 2024-09-22T03:16:19.099 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.099 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.099 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.100 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.100 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.100 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.100 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:16:19.100 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.795241+0000" 2024-09-22T03:16:19.100 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.100 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.100 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.100 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.100 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.100 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.100 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:16:19.100 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.100 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.100 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.101 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.101 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.101 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.101 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.101 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:16:19.101 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.833162+0000" 2024-09-22T03:16:19.101 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.101 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.101 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:16:19.101 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.795639+0000" 2024-09-22T03:16:19.101 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.101 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.101 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.102 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.102 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.102 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.102 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:16:19.102 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.102 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.102 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.102 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.102 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.102 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.102 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.102 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:16:19.102 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.796141+0000" 2024-09-22T03:16:19.102 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.103 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.103 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.103 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.103 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.103 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.103 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:16:19.103 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.103 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.103 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.103 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.103 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.103 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.103 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.103 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:16:19.104 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.796280+0000" 2024-09-22T03:16:19.104 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.104 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.104 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.104 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.104 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.104 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.104 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:16:19.104 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.104 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.104 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.104 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.104 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.104 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.104 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.105 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:16:19.105 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.105 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.105 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.105 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.105 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.105 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.105 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.105 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:16:19.105 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.105 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.105 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.105 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.105 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.105 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.106 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.106 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:16:19.106 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.796762+0000" 2024-09-22T03:16:19.106 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.106 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.106 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.106 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.106 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.106 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.106 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:16:19.106 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.106 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.106 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.107 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.107 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.107 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.107 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.107 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:16:19.107 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.797168+0000" 2024-09-22T03:16:19.107 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.107 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.107 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.107 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.107 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.107 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.108 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:16:19.108 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.108 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.108 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.108 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.108 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.108 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.108 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.108 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:16:19.108 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.831943+0000" 2024-09-22T03:16:19.108 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.108 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.108 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.108 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.109 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.109 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.109 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:16:19.109 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.109 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:16:19.109 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:16:19.109 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.109 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-22T03:16:19.109 INFO:teuthology.orchestra.run.smithi196.stdout: "type": "client_request", 2024-09-22T03:16:19.109 INFO:teuthology.orchestra.run.smithi196.stdout: "id": 317, 2024-09-22T03:16:19.109 INFO:teuthology.orchestra.run.smithi196.stdout: "detail": { 2024-09-22T03:16:19.109 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.109 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:16:19.109 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.796814+0000" 2024-09-22T03:16:19.109 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.110 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.110 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:16:19.110 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.796818+0000" 2024-09-22T03:16:19.110 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.110 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.110 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.110 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.110 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.110 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.110 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:16:19.110 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.796891+0000" 2024-09-22T03:16:19.110 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.110 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.110 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.111 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.111 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.111 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.111 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:16:19.111 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.111 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.111 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.111 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.111 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.111 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.111 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.111 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:16:19.111 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.797212+0000" 2024-09-22T03:16:19.111 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.111 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.112 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.112 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.112 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.112 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.112 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:16:19.112 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.797338+0000" 2024-09-22T03:16:19.112 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.112 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.112 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.112 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.112 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.112 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.112 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:16:19.112 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.113 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.113 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.113 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.113 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.113 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.113 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.113 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:16:19.113 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.840944+0000" 2024-09-22T03:16:19.113 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.113 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.113 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:16:19.113 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.797722+0000" 2024-09-22T03:16:19.113 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.113 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.114 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.114 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.114 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.114 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.114 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:16:19.114 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.114 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.114 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.114 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.114 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.114 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.114 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.114 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:16:19.114 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.799831+0000" 2024-09-22T03:16:19.115 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.115 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.115 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.115 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.115 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.115 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.115 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:16:19.115 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.115 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.115 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.115 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.115 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.115 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.116 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.116 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:16:19.116 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.800264+0000" 2024-09-22T03:16:19.116 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.116 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.116 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.116 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.116 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.116 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.116 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:16:19.116 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.116 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.116 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.117 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.117 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.117 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.117 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.117 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:16:19.117 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.117 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.117 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.117 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.117 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.117 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.117 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.118 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:16:19.118 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.118 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.118 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.118 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.118 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.118 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.118 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.118 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:16:19.118 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.800530+0000" 2024-09-22T03:16:19.118 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.119 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.119 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.119 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.119 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.119 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.119 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:16:19.119 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.120 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.120 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.120 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.120 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.120 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.120 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.120 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:16:19.121 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.801556+0000" 2024-09-22T03:16:19.121 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.121 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.121 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.121 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.121 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.121 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.121 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:16:19.121 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.121 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.121 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.121 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.121 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.121 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.122 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.122 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:16:19.122 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.839334+0000" 2024-09-22T03:16:19.122 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.122 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.122 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.122 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.122 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.122 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.122 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:16:19.122 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.122 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:16:19.123 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:16:19.123 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.123 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-22T03:16:19.123 INFO:teuthology.orchestra.run.smithi196.stdout: "type": "client_request", 2024-09-22T03:16:19.123 INFO:teuthology.orchestra.run.smithi196.stdout: "id": 319, 2024-09-22T03:16:19.123 INFO:teuthology.orchestra.run.smithi196.stdout: "detail": { 2024-09-22T03:16:19.123 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.123 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:16:19.123 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.805613+0000" 2024-09-22T03:16:19.123 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.123 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.123 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:16:19.123 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.805624+0000" 2024-09-22T03:16:19.123 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.123 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.124 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.124 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.124 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.124 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.124 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:16:19.124 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.806366+0000" 2024-09-22T03:16:19.124 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.124 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.124 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.124 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.124 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.124 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.124 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:16:19.125 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.125 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.125 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.125 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.125 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.125 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.125 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.125 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:16:19.125 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.806642+0000" 2024-09-22T03:16:19.125 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.125 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.125 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.125 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.126 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.126 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.126 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:16:19.126 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.807526+0000" 2024-09-22T03:16:19.126 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.126 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.126 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.126 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.126 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.126 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.126 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:16:19.126 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.126 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.126 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.127 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.127 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.127 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.127 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.127 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:16:19.127 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.855659+0000" 2024-09-22T03:16:19.127 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.127 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.127 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:16:19.127 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.807787+0000" 2024-09-22T03:16:19.127 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.128 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.128 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.128 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.128 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.128 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.128 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:16:19.128 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.128 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.128 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.128 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.128 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.128 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.128 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.128 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:16:19.128 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.808623+0000" 2024-09-22T03:16:19.129 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.129 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.129 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.129 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.129 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.129 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.129 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:16:19.129 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.129 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.129 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.129 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.129 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.129 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.129 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.130 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:16:19.130 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.808747+0000" 2024-09-22T03:16:19.130 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.130 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.130 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.130 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.130 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.130 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.130 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:16:19.130 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.130 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.130 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.130 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.130 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.131 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.131 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.131 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:16:19.131 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.131 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.131 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.131 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.131 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.131 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.131 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.131 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:16:19.131 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.131 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.131 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.132 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.132 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.132 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.132 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.132 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:16:19.132 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.808927+0000" 2024-09-22T03:16:19.132 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.132 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.132 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.132 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.133 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.133 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.133 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:16:19.133 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.133 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.133 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.133 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.134 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.134 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.134 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.134 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:16:19.134 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.809534+0000" 2024-09-22T03:16:19.134 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.134 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.135 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.135 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.135 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.135 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.135 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:16:19.135 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.136 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.136 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.136 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.136 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.136 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.136 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.136 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:16:19.137 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.851895+0000" 2024-09-22T03:16:19.137 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.137 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.137 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.137 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.137 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.138 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.138 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:16:19.138 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.138 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:16:19.138 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:16:19.138 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.138 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-22T03:16:19.139 INFO:teuthology.orchestra.run.smithi196.stdout: "type": "client_request", 2024-09-22T03:16:19.139 INFO:teuthology.orchestra.run.smithi196.stdout: "id": 320, 2024-09-22T03:16:19.139 INFO:teuthology.orchestra.run.smithi196.stdout: "detail": { 2024-09-22T03:16:19.139 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.139 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:16:19.139 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.809710+0000" 2024-09-22T03:16:19.140 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.140 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.140 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:16:19.140 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.809714+0000" 2024-09-22T03:16:19.140 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.140 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.140 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.141 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.141 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.141 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.141 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:16:19.141 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.810392+0000" 2024-09-22T03:16:19.141 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.142 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.142 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.142 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.142 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.142 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.142 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:16:19.143 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.143 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.143 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.143 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.143 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.143 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.144 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.144 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:16:19.144 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.812709+0000" 2024-09-22T03:16:19.144 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.144 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.144 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.144 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.145 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.145 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.145 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:16:19.145 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.813175+0000" 2024-09-22T03:16:19.145 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.145 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.146 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.146 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.146 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.146 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.146 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:16:19.146 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.146 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.147 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.147 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.147 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.147 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.147 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.147 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:16:19.148 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.859666+0000" 2024-09-22T03:16:19.148 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.148 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.148 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:16:19.148 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.813860+0000" 2024-09-22T03:16:19.148 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.148 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.149 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.149 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.149 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.149 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.149 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:16:19.149 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.150 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.150 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.150 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.150 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.150 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.150 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.150 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:16:19.151 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.814201+0000" 2024-09-22T03:16:19.151 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.151 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.151 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.151 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.151 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.152 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.152 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:16:19.152 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.152 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.152 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.152 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.153 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.153 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.153 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.153 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:16:19.153 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.814754+0000" 2024-09-22T03:16:19.153 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.154 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.154 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.154 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.154 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.154 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.154 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:16:19.154 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.155 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.155 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.155 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.155 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.155 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.155 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.156 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:16:19.156 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.156 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.156 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.156 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.156 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.156 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.157 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.157 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:16:19.157 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.157 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.157 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.157 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.157 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.158 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.158 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.158 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:16:19.158 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.814950+0000" 2024-09-22T03:16:19.158 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.158 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.158 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.159 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.159 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.159 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.159 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:16:19.159 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.159 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.159 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.159 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.159 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.159 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.159 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.159 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:16:19.160 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.816411+0000" 2024-09-22T03:16:19.160 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.160 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.160 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.160 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.160 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.160 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.160 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:16:19.160 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.160 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.160 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.160 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.160 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.160 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.161 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.161 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:16:19.161 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.858248+0000" 2024-09-22T03:16:19.161 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.161 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.161 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.161 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.161 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.161 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.161 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:16:19.161 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.161 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:16:19.161 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:16:19.161 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.161 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-22T03:16:19.162 INFO:teuthology.orchestra.run.smithi196.stdout: "type": "client_request", 2024-09-22T03:16:19.162 INFO:teuthology.orchestra.run.smithi196.stdout: "id": 321, 2024-09-22T03:16:19.162 INFO:teuthology.orchestra.run.smithi196.stdout: "detail": { 2024-09-22T03:16:19.162 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.162 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:16:19.162 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.815718+0000" 2024-09-22T03:16:19.162 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.162 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.162 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:16:19.162 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.815724+0000" 2024-09-22T03:16:19.162 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.162 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.162 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.163 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.163 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.163 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.163 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:16:19.163 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.815915+0000" 2024-09-22T03:16:19.163 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.163 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.163 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.163 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.163 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.163 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.163 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:16:19.163 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.164 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.164 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.164 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.164 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.164 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.164 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.164 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:16:19.164 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.816070+0000" 2024-09-22T03:16:19.164 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.164 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.164 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.164 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.164 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.164 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.165 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:16:19.165 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.816455+0000" 2024-09-22T03:16:19.165 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.165 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.165 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.165 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.165 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.165 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.165 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:16:19.165 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.165 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.165 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.165 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.165 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.165 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.166 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.166 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:16:19.166 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.866156+0000" 2024-09-22T03:16:19.166 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.166 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.166 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:16:19.166 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.818992+0000" 2024-09-22T03:16:19.166 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.166 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.166 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.166 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.166 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.166 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.166 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:16:19.167 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.167 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.167 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.167 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.167 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.167 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.167 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.167 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:16:19.167 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.819421+0000" 2024-09-22T03:16:19.167 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.167 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.167 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.167 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.167 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.167 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.168 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:16:19.168 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.168 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.168 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.168 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.168 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.168 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.168 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.168 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:16:19.168 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.819665+0000" 2024-09-22T03:16:19.168 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.168 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.168 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.168 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.169 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.169 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.169 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:16:19.169 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.169 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.169 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.169 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.169 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.169 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.169 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.169 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:16:19.169 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.169 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.169 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.170 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.170 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.170 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.170 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.170 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:16:19.170 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.170 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.170 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.170 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.170 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.170 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.170 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.170 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:16:19.170 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.819849+0000" 2024-09-22T03:16:19.171 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.171 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.171 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.171 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.171 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.171 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.171 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:16:19.171 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.171 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.171 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.171 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.171 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.171 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.171 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.172 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:16:19.172 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.820709+0000" 2024-09-22T03:16:19.172 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.172 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.172 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.172 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.172 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.172 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.172 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:16:19.172 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.172 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.172 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.172 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.172 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.172 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.173 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.173 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:16:19.173 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.865358+0000" 2024-09-22T03:16:19.173 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.173 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.173 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.173 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.173 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.173 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.173 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:16:19.173 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.173 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:16:19.173 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:16:19.173 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.174 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-22T03:16:19.174 INFO:teuthology.orchestra.run.smithi196.stdout: "type": "client_request", 2024-09-22T03:16:19.174 INFO:teuthology.orchestra.run.smithi196.stdout: "id": 322, 2024-09-22T03:16:19.174 INFO:teuthology.orchestra.run.smithi196.stdout: "detail": { 2024-09-22T03:16:19.174 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.174 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:16:19.174 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.820659+0000" 2024-09-22T03:16:19.174 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.174 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.174 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:16:19.174 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.820668+0000" 2024-09-22T03:16:19.174 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.174 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.174 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.175 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.175 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.175 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.175 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:16:19.175 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.821245+0000" 2024-09-22T03:16:19.175 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.175 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.175 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.175 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.175 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.175 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.175 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:16:19.175 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.175 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.176 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.176 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.176 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.176 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.176 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.176 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:16:19.176 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.821813+0000" 2024-09-22T03:16:19.176 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.176 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.176 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.176 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.176 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.176 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.176 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:16:19.177 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.823941+0000" 2024-09-22T03:16:19.177 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.177 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.177 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.177 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.177 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.177 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.177 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:16:19.177 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.177 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.177 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.177 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.177 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.177 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.177 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.178 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:16:19.178 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.868680+0000" 2024-09-22T03:16:19.178 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.178 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.178 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:16:19.178 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.824850+0000" 2024-09-22T03:16:19.178 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.178 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.178 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.178 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.178 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.178 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.178 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:16:19.178 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.179 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.179 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.179 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.179 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.179 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.179 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.179 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:16:19.179 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.825495+0000" 2024-09-22T03:16:19.179 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.179 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.179 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.179 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.179 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.179 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.180 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:16:19.180 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.180 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.180 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.180 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.180 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.180 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.180 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.180 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:16:19.180 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.827279+0000" 2024-09-22T03:16:19.180 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.180 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.180 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.180 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.180 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.181 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.181 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:16:19.181 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.181 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.181 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.181 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.181 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.181 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.181 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.181 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:16:19.181 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.181 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.181 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.181 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.182 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.182 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.182 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.182 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:16:19.182 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.182 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.182 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.182 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.182 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.182 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.182 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.182 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:16:19.182 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.827500+0000" 2024-09-22T03:16:19.182 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.183 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.183 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.183 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.183 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.183 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.183 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:16:19.183 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.183 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.183 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.183 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.183 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.183 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.183 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.183 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:16:19.184 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.827987+0000" 2024-09-22T03:16:19.184 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.184 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.184 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.184 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.184 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.184 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.184 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:16:19.184 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.184 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.184 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.184 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.184 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.184 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.185 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.185 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:16:19.185 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.867952+0000" 2024-09-22T03:16:19.185 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.185 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.185 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.185 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.185 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.185 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.185 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:16:19.185 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.185 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:16:19.185 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:16:19.186 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.186 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-22T03:16:19.186 INFO:teuthology.orchestra.run.smithi196.stdout: "type": "client_request", 2024-09-22T03:16:19.186 INFO:teuthology.orchestra.run.smithi196.stdout: "id": 323, 2024-09-22T03:16:19.186 INFO:teuthology.orchestra.run.smithi196.stdout: "detail": { 2024-09-22T03:16:19.186 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.186 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:16:19.186 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.830165+0000" 2024-09-22T03:16:19.186 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.186 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.186 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:16:19.186 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.830173+0000" 2024-09-22T03:16:19.186 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.186 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.187 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.187 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.187 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.187 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.187 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:16:19.187 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.830369+0000" 2024-09-22T03:16:19.187 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.187 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.187 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.187 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.187 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.187 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.187 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:16:19.187 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.187 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.188 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.188 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.188 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.188 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.188 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.188 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:16:19.188 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.830536+0000" 2024-09-22T03:16:19.188 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.188 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.188 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.188 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.188 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.188 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.188 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:16:19.189 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.830688+0000" 2024-09-22T03:16:19.189 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.189 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.189 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.189 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.189 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.189 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.189 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:16:19.189 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.189 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.189 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.189 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.189 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.189 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.190 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.190 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:16:19.190 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.871201+0000" 2024-09-22T03:16:19.190 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.190 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.190 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:16:19.190 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.831498+0000" 2024-09-22T03:16:19.190 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.190 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.190 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.190 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.190 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.191 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.191 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:16:19.191 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.191 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.191 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.191 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.191 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.191 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.191 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.191 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:16:19.191 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.831912+0000" 2024-09-22T03:16:19.191 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.192 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.192 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.192 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.192 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.192 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.192 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:16:19.192 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.192 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.192 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.192 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.192 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.192 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.192 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.192 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:16:19.193 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.832252+0000" 2024-09-22T03:16:19.193 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.193 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.193 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.193 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.193 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.193 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.193 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:16:19.193 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.193 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.193 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.193 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.193 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.194 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.194 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.194 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:16:19.194 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.194 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.194 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.194 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.194 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.194 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.194 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.194 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:16:19.194 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.194 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.195 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.195 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.195 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.195 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.195 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.195 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:16:19.195 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.833450+0000" 2024-09-22T03:16:19.195 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.195 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.195 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.195 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.195 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.195 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.196 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:16:19.196 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.196 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.196 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.196 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.196 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.196 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.196 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.196 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:16:19.196 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.833973+0000" 2024-09-22T03:16:19.196 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.196 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.197 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.197 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.197 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.197 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.197 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:16:19.197 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.197 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.197 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.197 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.197 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.197 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.197 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.197 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:16:19.197 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.869980+0000" 2024-09-22T03:16:19.198 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.198 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.198 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.198 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.198 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.198 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.198 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:16:19.198 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.198 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:16:19.198 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:16:19.198 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.198 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-22T03:16:19.198 INFO:teuthology.orchestra.run.smithi196.stdout: "type": "client_request", 2024-09-22T03:16:19.198 INFO:teuthology.orchestra.run.smithi196.stdout: "id": 324, 2024-09-22T03:16:19.199 INFO:teuthology.orchestra.run.smithi196.stdout: "detail": { 2024-09-22T03:16:19.199 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.199 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:16:19.199 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.836857+0000" 2024-09-22T03:16:19.199 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.199 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.199 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:16:19.199 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.836871+0000" 2024-09-22T03:16:19.199 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.200 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.200 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.200 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.200 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.200 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.200 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:16:19.200 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.837219+0000" 2024-09-22T03:16:19.200 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.200 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.200 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.200 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.200 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.200 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.200 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:16:19.201 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.201 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.201 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.201 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.201 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.201 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.201 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.201 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:16:19.201 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.837589+0000" 2024-09-22T03:16:19.201 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.201 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.201 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.201 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.201 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.202 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.202 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:16:19.202 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.837766+0000" 2024-09-22T03:16:19.202 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.202 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.202 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.202 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.202 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.202 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.202 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:16:19.202 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.202 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.202 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.202 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.203 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.203 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.203 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.203 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:16:19.203 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.873902+0000" 2024-09-22T03:16:19.203 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.203 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.203 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:16:19.203 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.838152+0000" 2024-09-22T03:16:19.203 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.203 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.203 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.203 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.203 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.204 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.204 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:16:19.204 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.204 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.204 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.204 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.204 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.204 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.204 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.204 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:16:19.204 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.838373+0000" 2024-09-22T03:16:19.204 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.204 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.204 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.205 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.205 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.205 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.205 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:16:19.205 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.205 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.205 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.205 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.205 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.205 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.205 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.205 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:16:19.205 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.838897+0000" 2024-09-22T03:16:19.205 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.206 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.206 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.206 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.206 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.206 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.206 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:16:19.206 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.206 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.206 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.206 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.206 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.206 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.206 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.207 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:16:19.207 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.207 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.207 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.207 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.207 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.207 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.207 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.207 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:16:19.207 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.207 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.207 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.207 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.207 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.208 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.208 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.208 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:16:19.208 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.839390+0000" 2024-09-22T03:16:19.208 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.208 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.208 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.208 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.208 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.208 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.208 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:16:19.208 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.208 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.208 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.209 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.209 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.209 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.209 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.209 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:16:19.209 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.840449+0000" 2024-09-22T03:16:19.209 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.209 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.209 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.209 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.209 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.209 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.210 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:16:19.210 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.210 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.210 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.210 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.210 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.210 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.210 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.210 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:16:19.210 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.873126+0000" 2024-09-22T03:16:19.210 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.210 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.210 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.210 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.211 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.211 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.211 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:16:19.211 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.211 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:16:19.211 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:16:19.211 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.211 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-22T03:16:19.211 INFO:teuthology.orchestra.run.smithi196.stdout: "type": "client_request", 2024-09-22T03:16:19.211 INFO:teuthology.orchestra.run.smithi196.stdout: "id": 325, 2024-09-22T03:16:19.211 INFO:teuthology.orchestra.run.smithi196.stdout: "detail": { 2024-09-22T03:16:19.211 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.211 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:16:19.212 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.840736+0000" 2024-09-22T03:16:19.212 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.212 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.212 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:16:19.212 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.840745+0000" 2024-09-22T03:16:19.212 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.212 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.212 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.212 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.212 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.212 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.212 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:16:19.212 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.841556+0000" 2024-09-22T03:16:19.212 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.213 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.213 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.213 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.213 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.213 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.213 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:16:19.213 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.213 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.213 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.213 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.213 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.213 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.213 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.213 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:16:19.213 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.841628+0000" 2024-09-22T03:16:19.214 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.214 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.214 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.214 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.214 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.214 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.214 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:16:19.214 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.843620+0000" 2024-09-22T03:16:19.214 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.214 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.214 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.214 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.214 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.215 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.215 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:16:19.215 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.215 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.215 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.215 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.215 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.215 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.215 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.215 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:16:19.215 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.875876+0000" 2024-09-22T03:16:19.215 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.215 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.215 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:16:19.216 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.844068+0000" 2024-09-22T03:16:19.216 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.216 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.216 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.216 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.216 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.216 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.216 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:16:19.216 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.216 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.216 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.216 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.216 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.216 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.217 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.217 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:16:19.217 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.844259+0000" 2024-09-22T03:16:19.217 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.217 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.217 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.217 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.217 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.217 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.217 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:16:19.217 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.217 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.217 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.217 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.217 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.218 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.218 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.218 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:16:19.218 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.844736+0000" 2024-09-22T03:16:19.218 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.218 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.218 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.218 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.218 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.218 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.218 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:16:19.218 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.218 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.218 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.219 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.219 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.219 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.219 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.219 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:16:19.219 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.219 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.219 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.219 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.219 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.219 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.219 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.219 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:16:19.220 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.220 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.220 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.220 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.220 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.220 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.220 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.220 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:16:19.220 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.845055+0000" 2024-09-22T03:16:19.220 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.220 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.220 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.221 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.221 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.221 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.221 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:16:19.221 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.221 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.221 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.221 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.221 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.221 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.221 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.221 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:16:19.221 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.845948+0000" 2024-09-22T03:16:19.221 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.221 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.222 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.222 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.222 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.222 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.222 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:16:19.222 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.222 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.222 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.222 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.222 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.222 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.222 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.222 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:16:19.222 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.875264+0000" 2024-09-22T03:16:19.223 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.223 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.223 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.223 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.223 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.223 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.223 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:16:19.223 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.223 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:16:19.223 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:16:19.223 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.223 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-22T03:16:19.223 INFO:teuthology.orchestra.run.smithi196.stdout: "type": "client_request", 2024-09-22T03:16:19.223 INFO:teuthology.orchestra.run.smithi196.stdout: "id": 326, 2024-09-22T03:16:19.224 INFO:teuthology.orchestra.run.smithi196.stdout: "detail": { 2024-09-22T03:16:19.224 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.224 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:16:19.224 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.846186+0000" 2024-09-22T03:16:19.224 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.224 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.224 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:16:19.224 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.846192+0000" 2024-09-22T03:16:19.224 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.224 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.224 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.224 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.224 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.225 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.225 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:16:19.225 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.847120+0000" 2024-09-22T03:16:19.225 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.225 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.225 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.225 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.225 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.225 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.225 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:16:19.225 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.225 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.225 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.225 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.226 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.226 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.226 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.226 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:16:19.226 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.848927+0000" 2024-09-22T03:16:19.226 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.226 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.226 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.226 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.226 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.226 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.226 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:16:19.226 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.849133+0000" 2024-09-22T03:16:19.226 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.227 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.227 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.227 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.227 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.227 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.227 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:16:19.227 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.227 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.227 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.227 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.227 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.227 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.227 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.227 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:16:19.227 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.876677+0000" 2024-09-22T03:16:19.228 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.228 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.228 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:16:19.228 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.849366+0000" 2024-09-22T03:16:19.228 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.228 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.229 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.229 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.229 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.229 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.229 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:16:19.230 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.230 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.230 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.230 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.230 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.230 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.230 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.230 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:16:19.230 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.849730+0000" 2024-09-22T03:16:19.230 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.230 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.230 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.231 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.231 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.231 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.231 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:16:19.231 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.231 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.231 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.231 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.231 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.231 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.231 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.231 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:16:19.231 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.850120+0000" 2024-09-22T03:16:19.231 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.232 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.232 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.232 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.232 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.232 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.232 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:16:19.232 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.232 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.232 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.232 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.232 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.232 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.232 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.233 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:16:19.233 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.233 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.233 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.233 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.233 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.233 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.233 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.233 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:16:19.233 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.233 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.233 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.233 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.233 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.233 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.234 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.234 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:16:19.234 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.850993+0000" 2024-09-22T03:16:19.234 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.234 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.234 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.234 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.234 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.234 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.234 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:16:19.234 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.234 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.235 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.235 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.235 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.235 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.235 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.235 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:16:19.235 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.851470+0000" 2024-09-22T03:16:19.235 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.235 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.235 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.235 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.235 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.236 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.236 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:16:19.236 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.236 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.236 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.236 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.236 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.236 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.236 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.236 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:16:19.236 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.876396+0000" 2024-09-22T03:16:19.236 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.236 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.236 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.237 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.237 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.237 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.237 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:16:19.237 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.237 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:16:19.237 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:16:19.237 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.237 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-22T03:16:19.237 INFO:teuthology.orchestra.run.smithi196.stdout: "type": "client_request", 2024-09-22T03:16:19.237 INFO:teuthology.orchestra.run.smithi196.stdout: "id": 328, 2024-09-22T03:16:19.237 INFO:teuthology.orchestra.run.smithi196.stdout: "detail": { 2024-09-22T03:16:19.237 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.237 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:16:19.238 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.855836+0000" 2024-09-22T03:16:19.238 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.238 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.238 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:16:19.238 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.855846+0000" 2024-09-22T03:16:19.238 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.238 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.238 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.238 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.238 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.238 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.238 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:16:19.238 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.856803+0000" 2024-09-22T03:16:19.238 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.239 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.239 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.239 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.239 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.239 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.239 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:16:19.239 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.239 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.239 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.239 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.239 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.239 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.239 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.239 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:16:19.240 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.856914+0000" 2024-09-22T03:16:19.240 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.240 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.240 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.240 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.240 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.240 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.240 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:16:19.240 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.856993+0000" 2024-09-22T03:16:19.240 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.240 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.240 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.240 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.241 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.241 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.241 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:16:19.241 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.241 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.241 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.241 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.241 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.241 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.241 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.241 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:16:19.241 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.878367+0000" 2024-09-22T03:16:19.241 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.241 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.242 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:16:19.242 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.857404+0000" 2024-09-22T03:16:19.242 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.242 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.242 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.242 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.242 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.242 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.242 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:16:19.242 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.242 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.242 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.242 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.242 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.243 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.243 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.243 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:16:19.243 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.857525+0000" 2024-09-22T03:16:19.243 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.243 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.243 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.243 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.243 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.243 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.243 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:16:19.243 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.243 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.243 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.244 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.244 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.244 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.244 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.244 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:16:19.244 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.857651+0000" 2024-09-22T03:16:19.244 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.244 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.244 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.244 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.244 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.244 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.244 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:16:19.244 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.245 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.245 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.245 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.245 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.245 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.245 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.245 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:16:19.245 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.245 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.245 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.245 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.245 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.245 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.245 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.245 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:16:19.246 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.246 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.246 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.246 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.246 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.246 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.246 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.246 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:16:19.246 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.857945+0000" 2024-09-22T03:16:19.246 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.246 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.246 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.246 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.247 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.247 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.247 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:16:19.247 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.247 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.247 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.247 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.247 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.247 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.247 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.247 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:16:19.247 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.858729+0000" 2024-09-22T03:16:19.247 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.247 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.248 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.248 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.248 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.248 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.248 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:16:19.248 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.248 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.248 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.248 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.248 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.248 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.248 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.248 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:16:19.248 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.877958+0000" 2024-09-22T03:16:19.249 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.249 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.249 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.249 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.249 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.249 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.249 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:16:19.249 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.249 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:16:19.249 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:16:19.249 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.249 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-22T03:16:19.249 INFO:teuthology.orchestra.run.smithi196.stdout: "type": "replicated_request", 2024-09-22T03:16:19.249 INFO:teuthology.orchestra.run.smithi196.stdout: "id": 349, 2024-09-22T03:16:19.249 INFO:teuthology.orchestra.run.smithi196.stdout: "detail": { 2024-09-22T03:16:19.250 INFO:teuthology.orchestra.run.smithi196.stdout: "RepRequest": { 2024-09-22T03:16:19.250 INFO:teuthology.orchestra.run.smithi196.stdout: "reqid": "client.4710.0:11109", 2024-09-22T03:16:19.250 INFO:teuthology.orchestra.run.smithi196.stdout: "pgid": "7.5", 2024-09-22T03:16:19.250 INFO:teuthology.orchestra.run.smithi196.stdout: "map_epoch": 102, 2024-09-22T03:16:19.250 INFO:teuthology.orchestra.run.smithi196.stdout: "min_epoch": 100, 2024-09-22T03:16:19.250 INFO:teuthology.orchestra.run.smithi196.stdout: "oid": "7:a6b350a6:::benchmark_data_smithi153_37783_object1388:head", 2024-09-22T03:16:19.250 INFO:teuthology.orchestra.run.smithi196.stdout: "from": "1" 2024-09-22T03:16:19.250 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:16:19.250 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:16:19.250 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.250 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-22T03:16:19.250 INFO:teuthology.orchestra.run.smithi196.stdout: "type": "replicated_request", 2024-09-22T03:16:19.251 INFO:teuthology.orchestra.run.smithi196.stdout: "id": 350, 2024-09-22T03:16:19.251 INFO:teuthology.orchestra.run.smithi196.stdout: "detail": { 2024-09-22T03:16:19.251 INFO:teuthology.orchestra.run.smithi196.stdout: "RepRequest": { 2024-09-22T03:16:19.251 INFO:teuthology.orchestra.run.smithi196.stdout: "reqid": "client.4710.0:11110", 2024-09-22T03:16:19.251 INFO:teuthology.orchestra.run.smithi196.stdout: "pgid": "7.5", 2024-09-22T03:16:19.251 INFO:teuthology.orchestra.run.smithi196.stdout: "map_epoch": 102, 2024-09-22T03:16:19.251 INFO:teuthology.orchestra.run.smithi196.stdout: "min_epoch": 100, 2024-09-22T03:16:19.251 INFO:teuthology.orchestra.run.smithi196.stdout: "oid": "7:a6b350a6:::benchmark_data_smithi153_37783_object1388:head", 2024-09-22T03:16:19.251 INFO:teuthology.orchestra.run.smithi196.stdout: "from": "1" 2024-09-22T03:16:19.251 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:16:19.251 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:16:19.251 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.251 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-22T03:16:19.251 INFO:teuthology.orchestra.run.smithi196.stdout: "type": "client_request", 2024-09-22T03:16:19.252 INFO:teuthology.orchestra.run.smithi196.stdout: "id": 189, 2024-09-22T03:16:19.252 INFO:teuthology.orchestra.run.smithi196.stdout: "detail": { 2024-09-22T03:16:19.252 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.252 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:16:19.252 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.156518+0000" 2024-09-22T03:16:19.252 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.252 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.252 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:16:19.252 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.156522+0000" 2024-09-22T03:16:19.252 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.252 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.252 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.252 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.252 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.252 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.253 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:16:19.253 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.156692+0000" 2024-09-22T03:16:19.253 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.253 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.253 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.253 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.253 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.253 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.253 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:16:19.253 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.253 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.253 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.253 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.253 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.254 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.254 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.254 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:16:19.254 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.157700+0000" 2024-09-22T03:16:19.254 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.254 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.254 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.254 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.254 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.254 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.254 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:16:19.254 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.159934+0000" 2024-09-22T03:16:19.254 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.254 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.255 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.255 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.255 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.255 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.255 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:16:19.255 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.255 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.255 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.255 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.255 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.255 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.255 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.255 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:16:19.255 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.205260+0000" 2024-09-22T03:16:19.255 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.256 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.256 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:16:19.256 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.160332+0000" 2024-09-22T03:16:19.256 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.256 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.256 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.256 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.256 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.256 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.256 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:16:19.256 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.257 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.257 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.257 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.257 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.257 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.257 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.257 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:16:19.257 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.160526+0000" 2024-09-22T03:16:19.257 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.257 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.257 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.257 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.257 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.258 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.258 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:16:19.258 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.258 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.258 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.258 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.258 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.258 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.258 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.258 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:16:19.258 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.160689+0000" 2024-09-22T03:16:19.258 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.258 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.259 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.259 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.259 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.259 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.259 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:16:19.259 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.259 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.259 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.259 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.259 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.259 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.259 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.259 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:16:19.259 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.259 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.259 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.260 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.260 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.260 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.260 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.260 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:16:19.260 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.260 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.260 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.260 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.260 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.260 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.260 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.260 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:16:19.260 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.160931+0000" 2024-09-22T03:16:19.261 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.261 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.261 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.261 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.261 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.261 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.261 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:16:19.261 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.261 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.261 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.261 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.262 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.262 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.262 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.262 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:16:19.262 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.161690+0000" 2024-09-22T03:16:19.262 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.262 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.262 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.262 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.262 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.262 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.262 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:16:19.262 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.263 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.263 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.263 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.263 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.263 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.263 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.263 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:16:19.263 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.204189+0000" 2024-09-22T03:16:19.263 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.263 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.263 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.263 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.263 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.264 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.264 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:16:19.264 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.264 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:16:19.264 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:16:19.264 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.264 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-22T03:16:19.264 INFO:teuthology.orchestra.run.smithi196.stdout: "type": "client_request", 2024-09-22T03:16:19.264 INFO:teuthology.orchestra.run.smithi196.stdout: "id": 190, 2024-09-22T03:16:19.264 INFO:teuthology.orchestra.run.smithi196.stdout: "detail": { 2024-09-22T03:16:19.264 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.264 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:16:19.264 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.172194+0000" 2024-09-22T03:16:19.264 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.265 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.265 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:16:19.265 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.172202+0000" 2024-09-22T03:16:19.265 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.265 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.265 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.265 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.265 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.265 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.265 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:16:19.265 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.173196+0000" 2024-09-22T03:16:19.265 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.265 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.266 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.266 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.266 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.266 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.266 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:16:19.266 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.266 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.266 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.266 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.266 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.266 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.266 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.266 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:16:19.267 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.175651+0000" 2024-09-22T03:16:19.267 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.267 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.267 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.267 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.267 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.267 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.267 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:16:19.267 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.176648+0000" 2024-09-22T03:16:19.267 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.267 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.267 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.267 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.267 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.268 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.268 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:16:19.268 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.268 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.268 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.268 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.268 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.268 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.268 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.268 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:16:19.268 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.218416+0000" 2024-09-22T03:16:19.268 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.268 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.268 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:16:19.269 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.176878+0000" 2024-09-22T03:16:19.269 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.269 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.269 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.269 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.269 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.269 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.269 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:16:19.269 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.269 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.269 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.269 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.269 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.270 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.270 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.270 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:16:19.270 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.176995+0000" 2024-09-22T03:16:19.270 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.270 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.270 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.270 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.270 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.270 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.270 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:16:19.270 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.270 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.270 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.271 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.271 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.271 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.271 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.271 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:16:19.271 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.177113+0000" 2024-09-22T03:16:19.271 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.271 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.271 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.271 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.271 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.271 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.272 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:16:19.272 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.272 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.272 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.272 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.272 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.272 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.272 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.272 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:16:19.272 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.273 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.273 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.273 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.273 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.273 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.273 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.273 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:16:19.273 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.273 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.273 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.273 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.273 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.273 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.274 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.274 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:16:19.274 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.177272+0000" 2024-09-22T03:16:19.274 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.274 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.274 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.274 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.274 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.274 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.274 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:16:19.274 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.274 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.274 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.274 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.275 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.275 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.275 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.275 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:16:19.275 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.177546+0000" 2024-09-22T03:16:19.275 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.275 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.275 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.275 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.275 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.275 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.275 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:16:19.275 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.275 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.276 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.276 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.276 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.276 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.276 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.276 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:16:19.276 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.217679+0000" 2024-09-22T03:16:19.276 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.276 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.276 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.276 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.276 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.276 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.276 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:16:19.277 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.277 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:16:19.277 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:16:19.277 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.277 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-22T03:16:19.277 INFO:teuthology.orchestra.run.smithi196.stdout: "type": "client_request", 2024-09-22T03:16:19.277 INFO:teuthology.orchestra.run.smithi196.stdout: "id": 192, 2024-09-22T03:16:19.277 INFO:teuthology.orchestra.run.smithi196.stdout: "detail": { 2024-09-22T03:16:19.277 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.277 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:16:19.277 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.184775+0000" 2024-09-22T03:16:19.277 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.277 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.277 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:16:19.278 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.184788+0000" 2024-09-22T03:16:19.278 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.278 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.278 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.278 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.278 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.278 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.278 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:16:19.278 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.185409+0000" 2024-09-22T03:16:19.278 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.278 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.278 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.278 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.278 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.279 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.279 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:16:19.279 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.279 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.279 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.279 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.279 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.279 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.279 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.279 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:16:19.279 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.185633+0000" 2024-09-22T03:16:19.279 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.279 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.279 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.279 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.280 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.280 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.280 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:16:19.280 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.186912+0000" 2024-09-22T03:16:19.280 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.280 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.280 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.280 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.280 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.280 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.280 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:16:19.280 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.280 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.280 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.281 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.281 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.281 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.281 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.281 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:16:19.281 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.223027+0000" 2024-09-22T03:16:19.281 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.281 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.281 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:16:19.281 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.187302+0000" 2024-09-22T03:16:19.281 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.281 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.282 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.282 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.282 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.282 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.282 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:16:19.282 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.282 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.282 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.282 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.282 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.282 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.282 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.282 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:16:19.282 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.187489+0000" 2024-09-22T03:16:19.283 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.283 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.283 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.283 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.283 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.283 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.283 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:16:19.283 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.283 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.283 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.283 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.283 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.283 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.283 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.284 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:16:19.284 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.187664+0000" 2024-09-22T03:16:19.284 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.284 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.284 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.284 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.284 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.284 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.284 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:16:19.284 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.284 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.284 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.284 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.284 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.285 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.285 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.285 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:16:19.285 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.285 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.285 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.285 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.285 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.285 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.285 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.285 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:16:19.285 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.285 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.285 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.286 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.286 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.286 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.286 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.286 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:16:19.286 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.187926+0000" 2024-09-22T03:16:19.286 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.286 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.286 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.286 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.286 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.286 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.286 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:16:19.286 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.287 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.287 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.287 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.287 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.287 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.287 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.287 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:16:19.287 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.188418+0000" 2024-09-22T03:16:19.287 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.287 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.287 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.287 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.288 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.288 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.288 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:16:19.288 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.288 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.288 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.288 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.288 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.288 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.288 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.288 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:16:19.288 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.222365+0000" 2024-09-22T03:16:19.288 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.288 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.289 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.289 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.289 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.289 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.289 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:16:19.289 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.289 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:16:19.289 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:16:19.289 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.289 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-22T03:16:19.289 INFO:teuthology.orchestra.run.smithi196.stdout: "type": "client_request", 2024-09-22T03:16:19.289 INFO:teuthology.orchestra.run.smithi196.stdout: "id": 193, 2024-09-22T03:16:19.289 INFO:teuthology.orchestra.run.smithi196.stdout: "detail": { 2024-09-22T03:16:19.289 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.290 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:16:19.290 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.187545+0000" 2024-09-22T03:16:19.290 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.290 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.290 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:16:19.290 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.187554+0000" 2024-09-22T03:16:19.290 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.290 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.290 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.290 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.290 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.290 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.290 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:16:19.291 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.187683+0000" 2024-09-22T03:16:19.291 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.291 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.291 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.291 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.291 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.291 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.291 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:16:19.291 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.291 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.291 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.292 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.292 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.292 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.292 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.292 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:16:19.292 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.187768+0000" 2024-09-22T03:16:19.292 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.292 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.292 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.292 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.292 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.292 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.292 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:16:19.292 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.190813+0000" 2024-09-22T03:16:19.293 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.293 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.293 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.293 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.293 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.293 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.293 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:16:19.293 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.293 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.293 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.293 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.293 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.293 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.294 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.294 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:16:19.294 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.226160+0000" 2024-09-22T03:16:19.294 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.294 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.294 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:16:19.294 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.191078+0000" 2024-09-22T03:16:19.294 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.294 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.294 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.294 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.294 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.294 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.294 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:16:19.295 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.295 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.295 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.295 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.295 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.295 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.295 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.295 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:16:19.295 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.191210+0000" 2024-09-22T03:16:19.295 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.295 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.295 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.295 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.295 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.296 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.296 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:16:19.296 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.296 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.296 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.296 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.296 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.296 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.296 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.296 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:16:19.296 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.191523+0000" 2024-09-22T03:16:19.296 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.296 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.297 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.297 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.297 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.297 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.297 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:16:19.297 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.297 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.297 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.297 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.297 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.297 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.297 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.297 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:16:19.297 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.297 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.298 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.298 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.298 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.298 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.298 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.298 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:16:19.298 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.298 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.298 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.298 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.298 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.298 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.298 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.298 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:16:19.299 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.191664+0000" 2024-09-22T03:16:19.299 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.299 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.299 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.299 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.299 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.299 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.299 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:16:19.299 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.299 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.299 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.299 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.299 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.300 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.300 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.300 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:16:19.300 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.192191+0000" 2024-09-22T03:16:19.300 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.300 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.300 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.300 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.300 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.300 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.300 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:16:19.300 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.300 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.301 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.301 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.301 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.301 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.301 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.301 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:16:19.301 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.223849+0000" 2024-09-22T03:16:19.301 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.301 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.301 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.301 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.301 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.301 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.302 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:16:19.302 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.302 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:16:19.302 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:16:19.302 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.302 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-22T03:16:19.302 INFO:teuthology.orchestra.run.smithi196.stdout: "type": "client_request", 2024-09-22T03:16:19.302 INFO:teuthology.orchestra.run.smithi196.stdout: "id": 194, 2024-09-22T03:16:19.302 INFO:teuthology.orchestra.run.smithi196.stdout: "detail": { 2024-09-22T03:16:19.302 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.302 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:16:19.302 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.188761+0000" 2024-09-22T03:16:19.303 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.303 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.303 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:16:19.303 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.188765+0000" 2024-09-22T03:16:19.303 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.303 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.303 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.303 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.303 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.303 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.303 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:16:19.304 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.189636+0000" 2024-09-22T03:16:19.304 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.304 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.304 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.304 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.304 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.304 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.304 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:16:19.304 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.304 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.304 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.304 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.304 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.305 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.305 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.305 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:16:19.305 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.190537+0000" 2024-09-22T03:16:19.305 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.305 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.305 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.305 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.305 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.305 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.305 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:16:19.305 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.190853+0000" 2024-09-22T03:16:19.305 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.305 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.305 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.306 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.306 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.306 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.306 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:16:19.306 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.306 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.306 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.306 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.306 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.306 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.306 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.306 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:16:19.306 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.227186+0000" 2024-09-22T03:16:19.307 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.307 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.307 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:16:19.307 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.191393+0000" 2024-09-22T03:16:19.307 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.307 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.307 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.307 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.307 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.307 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.307 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:16:19.307 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.307 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.307 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.308 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.308 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.308 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.308 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.308 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:16:19.308 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.191746+0000" 2024-09-22T03:16:19.308 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.308 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.308 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.308 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.308 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.308 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.308 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:16:19.308 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.309 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.309 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.309 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.309 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.309 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.309 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.309 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:16:19.309 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.191830+0000" 2024-09-22T03:16:19.309 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.309 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.309 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.309 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.309 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.309 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.309 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:16:19.310 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.310 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.310 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.310 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.310 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.310 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.310 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.310 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:16:19.310 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.310 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.310 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.310 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.310 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.310 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.311 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.311 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:16:19.311 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.311 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.311 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.311 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.311 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.311 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.311 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.311 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:16:19.311 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.192320+0000" 2024-09-22T03:16:19.311 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.311 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.311 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.312 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.312 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.312 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.312 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:16:19.312 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.312 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.312 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.312 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.312 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.192545+0000" 2024-09-22T03:16:19.312 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.312 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.312 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:16:19.313 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.192638+0000" 2024-09-22T03:16:19.313 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.313 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.313 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.313 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.313 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.313 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.313 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:16:19.313 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.313 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.313 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.313 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.313 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.313 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.314 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.314 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:16:19.314 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.226832+0000" 2024-09-22T03:16:19.314 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.314 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.314 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.314 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.314 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.314 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.314 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:16:19.314 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.314 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:16:19.314 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:16:19.315 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.315 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-22T03:16:19.315 INFO:teuthology.orchestra.run.smithi196.stdout: "type": "client_request", 2024-09-22T03:16:19.315 INFO:teuthology.orchestra.run.smithi196.stdout: "id": 196, 2024-09-22T03:16:19.315 INFO:teuthology.orchestra.run.smithi196.stdout: "detail": { 2024-09-22T03:16:19.315 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.315 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:16:19.315 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.191962+0000" 2024-09-22T03:16:19.315 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.315 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.315 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:16:19.315 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.191970+0000" 2024-09-22T03:16:19.315 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.315 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.316 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.316 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.316 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.316 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.316 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:16:19.316 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.192340+0000" 2024-09-22T03:16:19.316 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.316 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.316 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.316 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.316 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.316 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.316 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:16:19.316 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.317 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.317 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.317 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.317 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.317 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.317 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.317 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:16:19.317 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.193122+0000" 2024-09-22T03:16:19.317 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.317 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.317 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.317 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.317 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.317 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.318 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:16:19.318 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.195193+0000" 2024-09-22T03:16:19.318 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.318 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.318 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.318 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.318 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.318 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.318 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:16:19.318 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.318 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.318 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.318 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.318 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.319 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.319 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.319 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:16:19.319 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.227551+0000" 2024-09-22T03:16:19.319 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.319 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.319 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:16:19.319 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.195573+0000" 2024-09-22T03:16:19.319 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.319 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.319 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.319 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.319 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.319 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.320 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:16:19.320 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.320 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.320 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.320 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.320 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.320 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.320 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.320 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:16:19.320 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.195757+0000" 2024-09-22T03:16:19.320 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.320 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.320 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.320 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.320 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.321 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.321 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:16:19.321 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.321 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.321 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.321 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.321 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.321 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.321 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.321 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:16:19.321 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.195923+0000" 2024-09-22T03:16:19.321 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.321 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.322 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.322 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.322 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.322 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.322 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:16:19.322 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.322 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.322 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.322 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.322 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.322 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.322 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.323 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:16:19.323 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.323 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.323 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.323 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.323 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.323 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.323 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.323 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:16:19.323 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.323 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.323 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.323 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.323 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.324 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.324 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.324 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:16:19.324 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.196155+0000" 2024-09-22T03:16:19.324 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.324 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.324 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.324 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.324 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.324 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.324 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:16:19.324 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.324 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.324 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.324 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.325 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.196539+0000" 2024-09-22T03:16:19.325 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.325 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.325 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:16:19.325 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.196686+0000" 2024-09-22T03:16:19.325 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.325 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.325 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.325 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.325 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.325 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.325 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:16:19.325 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.325 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.326 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.326 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.326 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.226576+0000" 2024-09-22T03:16:19.326 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.326 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.326 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:16:19.326 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.226943+0000" 2024-09-22T03:16:19.326 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.326 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.326 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.326 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.326 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.326 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.327 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:16:19.327 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.327 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:16:19.327 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:16:19.327 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.327 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-22T03:16:19.327 INFO:teuthology.orchestra.run.smithi196.stdout: "type": "client_request", 2024-09-22T03:16:19.327 INFO:teuthology.orchestra.run.smithi196.stdout: "id": 198, 2024-09-22T03:16:19.327 INFO:teuthology.orchestra.run.smithi196.stdout: "detail": { 2024-09-22T03:16:19.327 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.327 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:16:19.327 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.194932+0000" 2024-09-22T03:16:19.327 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.327 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.328 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:16:19.328 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.194938+0000" 2024-09-22T03:16:19.328 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.328 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.328 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.328 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.328 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.328 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.328 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:16:19.328 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.195356+0000" 2024-09-22T03:16:19.328 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.328 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.328 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.328 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.328 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.328 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.329 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:16:19.329 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.329 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.329 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.329 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.329 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.329 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.329 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.329 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:16:19.329 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.195449+0000" 2024-09-22T03:16:19.329 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.329 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.329 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.329 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.329 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.330 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.330 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:16:19.330 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.196638+0000" 2024-09-22T03:16:19.330 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.330 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.330 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.330 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.330 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.330 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.330 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:16:19.330 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.330 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.330 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.330 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.330 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.331 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.331 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.331 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:16:19.331 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.234646+0000" 2024-09-22T03:16:19.331 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.331 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.331 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:16:19.331 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.197106+0000" 2024-09-22T03:16:19.331 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.331 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.331 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.331 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.331 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.331 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.331 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:16:19.331 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.332 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.332 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.332 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.332 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.332 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.332 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.332 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:16:19.332 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.197284+0000" 2024-09-22T03:16:19.332 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.332 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.332 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.332 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.332 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.333 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.333 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:16:19.333 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.333 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.333 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.333 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.333 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.333 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.333 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.333 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:16:19.333 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.199612+0000" 2024-09-22T03:16:19.334 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.334 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.334 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.334 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.334 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.334 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.334 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:16:19.334 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.334 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.334 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.334 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.334 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.334 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.334 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.334 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:16:19.335 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.335 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.335 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.335 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.335 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.335 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.335 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.335 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:16:19.335 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.335 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.335 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.335 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.335 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.336 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.336 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.336 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:16:19.336 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.199856+0000" 2024-09-22T03:16:19.336 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.336 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.336 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.336 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.336 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.336 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.336 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:16:19.336 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.336 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.336 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.337 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.337 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.200371+0000" 2024-09-22T03:16:19.337 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.337 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.337 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:16:19.337 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.201681+0000" 2024-09-22T03:16:19.337 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.337 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.337 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.337 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.337 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.337 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.337 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:16:19.337 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.338 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.338 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.338 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.338 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.338 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.338 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.338 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:16:19.338 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.234260+0000" 2024-09-22T03:16:19.338 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.338 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.338 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.338 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.338 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.338 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.339 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:16:19.339 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.339 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:16:19.339 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:16:19.339 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.339 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-22T03:16:19.339 INFO:teuthology.orchestra.run.smithi196.stdout: "type": "client_request", 2024-09-22T03:16:19.339 INFO:teuthology.orchestra.run.smithi196.stdout: "id": 199, 2024-09-22T03:16:19.339 INFO:teuthology.orchestra.run.smithi196.stdout: "detail": { 2024-09-22T03:16:19.339 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.339 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:16:19.339 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.199323+0000" 2024-09-22T03:16:19.339 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.340 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.340 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:16:19.340 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.199329+0000" 2024-09-22T03:16:19.340 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.340 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.340 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.340 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.340 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.340 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.340 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:16:19.340 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.200002+0000" 2024-09-22T03:16:19.340 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.340 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.340 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.341 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.341 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.341 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.341 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:16:19.341 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.341 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.341 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.341 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.341 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.341 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.341 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.341 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:16:19.341 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.200539+0000" 2024-09-22T03:16:19.341 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.342 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.342 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.342 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.342 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.342 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.342 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:16:19.342 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.203353+0000" 2024-09-22T03:16:19.342 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.342 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.342 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.342 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.342 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.342 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.342 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:16:19.343 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.343 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.343 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.343 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.343 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.343 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.343 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.343 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:16:19.343 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.239750+0000" 2024-09-22T03:16:19.343 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.343 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.343 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:16:19.344 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.204000+0000" 2024-09-22T03:16:19.344 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.344 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.344 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.344 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.344 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.344 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.344 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:16:19.344 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.344 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.344 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.344 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.344 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.344 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.345 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.345 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:16:19.345 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.204168+0000" 2024-09-22T03:16:19.345 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.345 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.345 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.345 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.345 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.345 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.345 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:16:19.345 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.345 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.345 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.346 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.346 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.346 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.346 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.346 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:16:19.346 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.204424+0000" 2024-09-22T03:16:19.346 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.346 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.346 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.346 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.346 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.346 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.347 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:16:19.347 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.347 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.347 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.347 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.347 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.347 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.347 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.347 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:16:19.347 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.347 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.347 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.347 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.347 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.347 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.348 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.348 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:16:19.348 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.348 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.348 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.348 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.348 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.348 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.348 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.348 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:16:19.348 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.205057+0000" 2024-09-22T03:16:19.348 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.348 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.348 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.349 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.349 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.349 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.349 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:16:19.349 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.349 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.349 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.349 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.349 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.205494+0000" 2024-09-22T03:16:19.349 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.349 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.349 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:16:19.349 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.206456+0000" 2024-09-22T03:16:19.350 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.350 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.350 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.350 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.350 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.350 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.350 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:16:19.350 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.350 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.350 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.350 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.350 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.350 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.350 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.351 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:16:19.351 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.235164+0000" 2024-09-22T03:16:19.351 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.351 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.351 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.351 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.351 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.351 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.351 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:16:19.351 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.351 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:16:19.351 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:16:19.351 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.351 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-22T03:16:19.352 INFO:teuthology.orchestra.run.smithi196.stdout: "type": "client_request", 2024-09-22T03:16:19.352 INFO:teuthology.orchestra.run.smithi196.stdout: "id": 200, 2024-09-22T03:16:19.352 INFO:teuthology.orchestra.run.smithi196.stdout: "detail": { 2024-09-22T03:16:19.352 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.352 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:16:19.352 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.202825+0000" 2024-09-22T03:16:19.352 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.352 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.352 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:16:19.352 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.202832+0000" 2024-09-22T03:16:19.352 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.352 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.352 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.352 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.353 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.353 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.353 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:16:19.353 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.202954+0000" 2024-09-22T03:16:19.353 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.353 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.353 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.353 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.353 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.353 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.353 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:16:19.353 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.354 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.354 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.354 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.354 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.354 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.354 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.354 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:16:19.354 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.203050+0000" 2024-09-22T03:16:19.354 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.354 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.354 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.354 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.354 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.354 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.354 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:16:19.355 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.204332+0000" 2024-09-22T03:16:19.355 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.355 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.355 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.355 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.355 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.355 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.355 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:16:19.355 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.355 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.355 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.355 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.355 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.356 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.356 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.356 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:16:19.356 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.240766+0000" 2024-09-22T03:16:19.356 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.356 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.356 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:16:19.356 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.204966+0000" 2024-09-22T03:16:19.356 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.356 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.356 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.356 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.356 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.356 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.356 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:16:19.357 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.357 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.357 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.357 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.357 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.357 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.357 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.357 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:16:19.357 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.205555+0000" 2024-09-22T03:16:19.357 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.357 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.357 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.357 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.357 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.358 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.358 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:16:19.358 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.358 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.358 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.358 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.358 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.358 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.358 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.359 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:16:19.359 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.205686+0000" 2024-09-22T03:16:19.359 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.359 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.359 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.359 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.359 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.360 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.360 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:16:19.360 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.360 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.360 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.360 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.360 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.361 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.361 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.361 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:16:19.361 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.361 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.361 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.361 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.362 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.362 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.362 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.362 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:16:19.362 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.362 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.363 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.363 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.363 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.363 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.363 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.363 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:16:19.363 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.205850+0000" 2024-09-22T03:16:19.364 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.364 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.364 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.364 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.364 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.364 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.365 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:16:19.365 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.365 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.365 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.365 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.365 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.209619+0000" 2024-09-22T03:16:19.365 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.366 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.366 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:16:19.366 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.210465+0000" 2024-09-22T03:16:19.366 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.366 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.366 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.367 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.367 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.367 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.367 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:16:19.367 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.367 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.368 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.368 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.368 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.368 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.368 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.368 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:16:19.369 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.240254+0000" 2024-09-22T03:16:19.369 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.369 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.369 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.369 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.369 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.369 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.370 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:16:19.370 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.370 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:16:19.370 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:16:19.370 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.370 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-22T03:16:19.370 INFO:teuthology.orchestra.run.smithi196.stdout: "type": "client_request", 2024-09-22T03:16:19.371 INFO:teuthology.orchestra.run.smithi196.stdout: "id": 201, 2024-09-22T03:16:19.371 INFO:teuthology.orchestra.run.smithi196.stdout: "detail": { 2024-09-22T03:16:19.371 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.371 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:16:19.371 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.206947+0000" 2024-09-22T03:16:19.372 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.372 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.372 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:16:19.372 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.206953+0000" 2024-09-22T03:16:19.372 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.372 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.373 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.373 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.373 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.373 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.373 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:16:19.373 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.207027+0000" 2024-09-22T03:16:19.374 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.374 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.374 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.374 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.374 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.374 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.374 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:16:19.375 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.375 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.375 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.375 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.375 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.375 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.375 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.376 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:16:19.376 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.207193+0000" 2024-09-22T03:16:19.376 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.376 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.376 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.376 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.377 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.377 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.377 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:16:19.377 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.208266+0000" 2024-09-22T03:16:19.377 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.377 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.378 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.378 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.378 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.378 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.378 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:16:19.378 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.379 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.379 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.379 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.379 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.379 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.379 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.380 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:16:19.380 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.241103+0000" 2024-09-22T03:16:19.380 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.380 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.380 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:16:19.380 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.208642+0000" 2024-09-22T03:16:19.380 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.381 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.381 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.381 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.381 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.381 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.381 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:16:19.382 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.382 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.382 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.382 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.382 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.382 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.382 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.383 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:16:19.383 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.208814+0000" 2024-09-22T03:16:19.383 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.383 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.383 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.383 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.383 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.384 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.384 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:16:19.384 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.384 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.384 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.384 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.384 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.384 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.384 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.385 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:16:19.385 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.208941+0000" 2024-09-22T03:16:19.385 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.385 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.385 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.385 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.385 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.385 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.385 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:16:19.385 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.385 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.385 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.385 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.385 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.386 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.386 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.386 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:16:19.386 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.386 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.386 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.386 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.386 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.386 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.386 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.386 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:16:19.386 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.386 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.386 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.387 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.387 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.387 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.387 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.387 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:16:19.387 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.209164+0000" 2024-09-22T03:16:19.387 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.387 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.387 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.387 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.387 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.387 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.387 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:16:19.387 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.388 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.388 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.388 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.388 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.209860+0000" 2024-09-22T03:16:19.388 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.388 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.388 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:16:19.388 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.214500+0000" 2024-09-22T03:16:19.388 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.388 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.388 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.388 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.388 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.388 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.389 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:16:19.389 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.389 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.389 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.389 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.389 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.240246+0000" 2024-09-22T03:16:19.389 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.389 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.389 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:16:19.389 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.240517+0000" 2024-09-22T03:16:19.389 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.389 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.389 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.389 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.389 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.390 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.390 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:16:19.390 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.390 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:16:19.390 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:16:19.390 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.390 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-22T03:16:19.390 INFO:teuthology.orchestra.run.smithi196.stdout: "type": "client_request", 2024-09-22T03:16:19.390 INFO:teuthology.orchestra.run.smithi196.stdout: "id": 202, 2024-09-22T03:16:19.390 INFO:teuthology.orchestra.run.smithi196.stdout: "detail": { 2024-09-22T03:16:19.390 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.390 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:16:19.390 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.208282+0000" 2024-09-22T03:16:19.390 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.391 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.391 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:16:19.391 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.208286+0000" 2024-09-22T03:16:19.391 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.391 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.391 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.391 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.391 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.391 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.391 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:16:19.391 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.209016+0000" 2024-09-22T03:16:19.391 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.391 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.392 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.392 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.392 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.392 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.392 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:16:19.392 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.392 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.392 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.392 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.392 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.392 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.392 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.392 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:16:19.392 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.209110+0000" 2024-09-22T03:16:19.392 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.393 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.393 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.393 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.393 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.393 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.393 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:16:19.393 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.209929+0000" 2024-09-22T03:16:19.393 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.393 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.393 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.393 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.393 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.393 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.393 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:16:19.394 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.394 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.394 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.394 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.394 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.394 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.394 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.394 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:16:19.394 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.243885+0000" 2024-09-22T03:16:19.394 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.394 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.394 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:16:19.394 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.210152+0000" 2024-09-22T03:16:19.395 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.395 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.395 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.395 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.395 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.395 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.395 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:16:19.395 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.395 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.395 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.395 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.395 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.395 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.395 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.396 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:16:19.396 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.210252+0000" 2024-09-22T03:16:19.396 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.396 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.396 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.396 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.396 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.396 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.396 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:16:19.396 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.396 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.396 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.396 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.396 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.396 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.397 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.397 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:16:19.397 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.210550+0000" 2024-09-22T03:16:19.397 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.397 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.397 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.397 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.397 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.397 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.397 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:16:19.398 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.398 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.398 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.398 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.398 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.398 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.398 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.398 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:16:19.398 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.398 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.398 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.398 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.398 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.399 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.399 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.399 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:16:19.399 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.399 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.399 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.399 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.399 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.399 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.399 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.399 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:16:19.399 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.210967+0000" 2024-09-22T03:16:19.399 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.399 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.400 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.400 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.400 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.400 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.400 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:16:19.400 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.400 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.400 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.400 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.400 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.212817+0000" 2024-09-22T03:16:19.400 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.400 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.400 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:16:19.401 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.219240+0000" 2024-09-22T03:16:19.401 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.401 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.401 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.401 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.401 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.401 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.402 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:16:19.402 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.402 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.402 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.402 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.402 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.402 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.402 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.402 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:16:19.402 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.243530+0000" 2024-09-22T03:16:19.402 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.402 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.402 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.403 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.403 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.403 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.403 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:16:19.403 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.403 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:16:19.403 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:16:19.403 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.403 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-22T03:16:19.403 INFO:teuthology.orchestra.run.smithi196.stdout: "type": "client_request", 2024-09-22T03:16:19.403 INFO:teuthology.orchestra.run.smithi196.stdout: "id": 203, 2024-09-22T03:16:19.403 INFO:teuthology.orchestra.run.smithi196.stdout: "detail": { 2024-09-22T03:16:19.403 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.403 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:16:19.404 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.212095+0000" 2024-09-22T03:16:19.404 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.404 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.404 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:16:19.404 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.212100+0000" 2024-09-22T03:16:19.404 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.404 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.404 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.404 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.404 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.404 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.404 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:16:19.404 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.212183+0000" 2024-09-22T03:16:19.404 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.405 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.405 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.405 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.405 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.405 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.405 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:16:19.405 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.405 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.405 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.405 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.405 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.405 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.405 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.405 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:16:19.405 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.212395+0000" 2024-09-22T03:16:19.406 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.406 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.406 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.406 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.406 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.406 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.406 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:16:19.406 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.212929+0000" 2024-09-22T03:16:19.406 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.406 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.406 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.406 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.406 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.407 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.407 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:16:19.407 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.407 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.407 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.407 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.407 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.407 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.407 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.407 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:16:19.407 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.249392+0000" 2024-09-22T03:16:19.407 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.408 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.408 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:16:19.408 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.213282+0000" 2024-09-22T03:16:19.408 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.408 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.408 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.408 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.408 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.408 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.408 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:16:19.409 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.409 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.409 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.409 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.409 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.409 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.409 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.409 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:16:19.409 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.213451+0000" 2024-09-22T03:16:19.409 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.409 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.409 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.409 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.409 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.410 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.410 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:16:19.410 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.410 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.410 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.410 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.410 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.410 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.410 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.410 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:16:19.410 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.213599+0000" 2024-09-22T03:16:19.410 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.410 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.410 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.411 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.411 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.411 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.411 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:16:19.411 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.411 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.411 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.411 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.411 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.411 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.411 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.411 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:16:19.411 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.411 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.412 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.412 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.412 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.412 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.412 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.413 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:16:19.413 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.413 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.413 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.413 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.414 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.414 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.414 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.414 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:16:19.414 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.213860+0000" 2024-09-22T03:16:19.414 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.414 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.414 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.414 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.414 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.414 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.414 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:16:19.414 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.415 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.415 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.415 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.415 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.214188+0000" 2024-09-22T03:16:19.415 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.415 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.415 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:16:19.415 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.223827+0000" 2024-09-22T03:16:19.415 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.415 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.415 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.416 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.416 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.416 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.416 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:16:19.416 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.416 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.417 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.417 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.417 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.417 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.417 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.417 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:16:19.418 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.248997+0000" 2024-09-22T03:16:19.418 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.418 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.418 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.418 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.418 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.419 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.419 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:16:19.419 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.419 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:16:19.419 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:16:19.419 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.420 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-22T03:16:19.420 INFO:teuthology.orchestra.run.smithi196.stdout: "type": "client_request", 2024-09-22T03:16:19.420 INFO:teuthology.orchestra.run.smithi196.stdout: "id": 204, 2024-09-22T03:16:19.420 INFO:teuthology.orchestra.run.smithi196.stdout: "detail": { 2024-09-22T03:16:19.420 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.420 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:16:19.420 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.212739+0000" 2024-09-22T03:16:19.420 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.420 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.420 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:16:19.420 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.212743+0000" 2024-09-22T03:16:19.421 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.421 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.421 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.421 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.421 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.421 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.421 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:16:19.421 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.212792+0000" 2024-09-22T03:16:19.421 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.421 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.421 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.421 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.422 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.422 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.422 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:16:19.422 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.422 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.422 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.422 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.422 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.422 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.422 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.422 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:16:19.422 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.212843+0000" 2024-09-22T03:16:19.422 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.423 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.423 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.423 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.423 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.423 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.423 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:16:19.423 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.214250+0000" 2024-09-22T03:16:19.423 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.423 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.423 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.423 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.423 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.423 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.423 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:16:19.423 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.424 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.424 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.424 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.424 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.424 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.424 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.424 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:16:19.424 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.255959+0000" 2024-09-22T03:16:19.424 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.424 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.424 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:16:19.424 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.214703+0000" 2024-09-22T03:16:19.424 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.424 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.425 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.425 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.425 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.425 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.425 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:16:19.425 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.425 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.425 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.426 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.426 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.426 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.426 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.426 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:16:19.426 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.214875+0000" 2024-09-22T03:16:19.426 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.426 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.426 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.426 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.426 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.426 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.426 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:16:19.427 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.427 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.427 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.427 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.427 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.427 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.427 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.427 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:16:19.427 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.216542+0000" 2024-09-22T03:16:19.427 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.427 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.427 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.427 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.427 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.428 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.428 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:16:19.428 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.428 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.428 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.428 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.428 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.428 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.428 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.428 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:16:19.428 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.428 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.428 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.428 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.429 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.429 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.429 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.429 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:16:19.429 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.429 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.429 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.429 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.429 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.429 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.429 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.429 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:16:19.429 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.216798+0000" 2024-09-22T03:16:19.429 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.430 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.430 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.430 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.430 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.430 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.430 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:16:19.430 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.430 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.430 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.430 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.430 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.217241+0000" 2024-09-22T03:16:19.430 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.430 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.431 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:16:19.431 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.232098+0000" 2024-09-22T03:16:19.431 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.431 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.431 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.431 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.431 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.431 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.431 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:16:19.431 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.431 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.431 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.431 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.431 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.432 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.432 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.432 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:16:19.432 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.255669+0000" 2024-09-22T03:16:19.432 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.432 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.432 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.432 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.432 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.432 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.432 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:16:19.432 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.432 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:16:19.433 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:16:19.433 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.433 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-22T03:16:19.433 INFO:teuthology.orchestra.run.smithi196.stdout: "type": "client_request", 2024-09-22T03:16:19.433 INFO:teuthology.orchestra.run.smithi196.stdout: "id": 205, 2024-09-22T03:16:19.433 INFO:teuthology.orchestra.run.smithi196.stdout: "detail": { 2024-09-22T03:16:19.433 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.433 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:16:19.433 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.214376+0000" 2024-09-22T03:16:19.433 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.434 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.434 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:16:19.434 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.214382+0000" 2024-09-22T03:16:19.434 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.434 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.434 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.434 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.434 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.434 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.434 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:16:19.434 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.214528+0000" 2024-09-22T03:16:19.434 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.434 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.435 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.435 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.435 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.435 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.435 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:16:19.435 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.435 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.435 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.435 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.435 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.435 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.436 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.436 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:16:19.436 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.214747+0000" 2024-09-22T03:16:19.436 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.436 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.436 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.436 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.436 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.436 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.436 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:16:19.436 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.217388+0000" 2024-09-22T03:16:19.436 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.436 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.436 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.436 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.436 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.437 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.437 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:16:19.437 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.437 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.437 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.437 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.437 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.437 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.437 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.437 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:16:19.437 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.267429+0000" 2024-09-22T03:16:19.437 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.437 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.437 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:16:19.438 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.218071+0000" 2024-09-22T03:16:19.438 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.438 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.438 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.438 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.438 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.438 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.438 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:16:19.438 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.438 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.438 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.438 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.438 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.439 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.439 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.439 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:16:19.439 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.218276+0000" 2024-09-22T03:16:19.439 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.439 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.439 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.439 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.439 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.440 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.440 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:16:19.440 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.440 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.440 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.440 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.440 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.441 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.441 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.441 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:16:19.441 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.218543+0000" 2024-09-22T03:16:19.441 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.442 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.442 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.442 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.442 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.442 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.442 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:16:19.442 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.443 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.443 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.443 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.443 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.443 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.443 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.444 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:16:19.444 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.444 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.444 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.444 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.445 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.445 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.445 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.445 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:16:19.445 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.445 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.445 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.446 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.446 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.446 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.446 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.446 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:16:19.446 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.218669+0000" 2024-09-22T03:16:19.446 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.446 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.446 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.446 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.446 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.446 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.446 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:16:19.447 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.447 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.447 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.447 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.447 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.218897+0000" 2024-09-22T03:16:19.447 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.447 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.447 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:16:19.447 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.235305+0000" 2024-09-22T03:16:19.447 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.447 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.447 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.447 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.448 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.448 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.448 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:16:19.448 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.448 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.448 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.448 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.448 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.449 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.449 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.449 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:16:19.449 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.267203+0000" 2024-09-22T03:16:19.449 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.449 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.449 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.449 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.449 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.449 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.449 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:16:19.449 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.449 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:16:19.450 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:16:19.450 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.450 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-22T03:16:19.450 INFO:teuthology.orchestra.run.smithi196.stdout: "type": "client_request", 2024-09-22T03:16:19.450 INFO:teuthology.orchestra.run.smithi196.stdout: "id": 206, 2024-09-22T03:16:19.450 INFO:teuthology.orchestra.run.smithi196.stdout: "detail": { 2024-09-22T03:16:19.450 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.450 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:16:19.450 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.216950+0000" 2024-09-22T03:16:19.450 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.450 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.450 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:16:19.450 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.216959+0000" 2024-09-22T03:16:19.450 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.451 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.451 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.451 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.451 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.451 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.451 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:16:19.451 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.217120+0000" 2024-09-22T03:16:19.451 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.451 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.451 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.451 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.451 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.451 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.452 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:16:19.452 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.452 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.452 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.452 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.452 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.452 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.452 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.452 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:16:19.452 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.217413+0000" 2024-09-22T03:16:19.452 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.452 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.452 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.452 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.453 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.453 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.453 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:16:19.453 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.218954+0000" 2024-09-22T03:16:19.453 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.453 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.453 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.453 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.453 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.453 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.453 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:16:19.453 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.453 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.453 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.453 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.454 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.454 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.454 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.454 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:16:19.454 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.277206+0000" 2024-09-22T03:16:19.454 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.454 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.454 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:16:19.454 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.219185+0000" 2024-09-22T03:16:19.454 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.454 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.454 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.454 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.454 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.455 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.455 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:16:19.455 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.455 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.455 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.455 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.455 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.455 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.455 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.455 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:16:19.455 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.221148+0000" 2024-09-22T03:16:19.455 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.455 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.456 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.456 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.456 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.456 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.456 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:16:19.456 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.456 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.456 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.456 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.456 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.456 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.456 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.457 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:16:19.457 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.221308+0000" 2024-09-22T03:16:19.457 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.457 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.457 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.457 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.457 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.457 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.457 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:16:19.458 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.458 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.458 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.458 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.458 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.458 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.458 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.458 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:16:19.458 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.458 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.458 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.458 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.458 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.459 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.459 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.459 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:16:19.459 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.459 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.459 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.459 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.459 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.459 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.459 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.459 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:16:19.459 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.221522+0000" 2024-09-22T03:16:19.459 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.459 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.460 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.460 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.460 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.460 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.460 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:16:19.460 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.460 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.460 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.460 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.460 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.221923+0000" 2024-09-22T03:16:19.460 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.460 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.460 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:16:19.460 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.241116+0000" 2024-09-22T03:16:19.461 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.461 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.461 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.461 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.461 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.461 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.461 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:16:19.461 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.461 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.461 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.461 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.461 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.461 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.461 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.461 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:16:19.462 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.275298+0000" 2024-09-22T03:16:19.462 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.462 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.462 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.462 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.462 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.462 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.462 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:16:19.462 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.462 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:16:19.462 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:16:19.462 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.462 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-22T03:16:19.462 INFO:teuthology.orchestra.run.smithi196.stdout: "type": "client_request", 2024-09-22T03:16:19.462 INFO:teuthology.orchestra.run.smithi196.stdout: "id": 207, 2024-09-22T03:16:19.463 INFO:teuthology.orchestra.run.smithi196.stdout: "detail": { 2024-09-22T03:16:19.463 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.463 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:16:19.463 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.220454+0000" 2024-09-22T03:16:19.463 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.463 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.463 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:16:19.463 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.220462+0000" 2024-09-22T03:16:19.463 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.463 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.463 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.463 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.463 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.463 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.464 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:16:19.464 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.220739+0000" 2024-09-22T03:16:19.464 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.464 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.464 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.464 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.464 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.464 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.464 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:16:19.464 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.464 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.464 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.464 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.464 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.464 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.464 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.465 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:16:19.465 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.220913+0000" 2024-09-22T03:16:19.465 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.465 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.465 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.465 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.465 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.465 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.466 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:16:19.466 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.222058+0000" 2024-09-22T03:16:19.466 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.466 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.466 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.466 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.466 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.466 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.467 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:16:19.467 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.467 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.467 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.467 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.467 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.467 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.467 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.467 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:16:19.467 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.291213+0000" 2024-09-22T03:16:19.467 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.467 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.467 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:16:19.467 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.222654+0000" 2024-09-22T03:16:19.468 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.468 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.468 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.468 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.468 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.468 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.468 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:16:19.468 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.468 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.468 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.468 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.468 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.468 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.468 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.469 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:16:19.469 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.223085+0000" 2024-09-22T03:16:19.469 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.469 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.469 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.469 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.469 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.469 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.469 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:16:19.469 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.469 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.469 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.469 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.469 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.470 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.470 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.470 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:16:19.470 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.223214+0000" 2024-09-22T03:16:19.470 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.470 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.470 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.470 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.470 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.470 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.470 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:16:19.470 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.471 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.471 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.471 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.471 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.471 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.471 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.471 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:16:19.471 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.471 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.471 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.471 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.472 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.472 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.472 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.472 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:16:19.472 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.472 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.472 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.472 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.472 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.473 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.473 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.473 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:16:19.473 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.223339+0000" 2024-09-22T03:16:19.473 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.473 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.473 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.473 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.473 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.473 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.473 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:16:19.474 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.474 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.474 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.474 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.474 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.223576+0000" 2024-09-22T03:16:19.474 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.474 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.474 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:16:19.474 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.246295+0000" 2024-09-22T03:16:19.474 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.474 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.474 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.475 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.475 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.475 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.475 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:16:19.475 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.475 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.475 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.475 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.475 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.475 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.475 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:16:19.475 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:16:19.475 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:16:18.290961+0000" 2024-09-22T03:16:19.475 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.476 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.476 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:16:19.476 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.476 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:16:19.476 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:16:19.476 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:16:19.476 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:16:19.476 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:16:19.476 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:16:19.476 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:16:19.476 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-22T03:16:19.476 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-09-22T03:16:19.477 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 58 16 11117 11101 1.49514 0.094122 0.104717 0.0835279 2024-09-22T03:16:19.478 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-09-22T03:16:19.604 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:16:19.596+0000 7f0a12c6c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:16:19.608 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:16:19.607+0000 7f0a12c6c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:16:19.939 INFO:teuthology.orchestra.run.smithi153.stderr:reweighted osd.0 to 1 (10000) 2024-09-22T03:16:19.960 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-09-22T03:16:19.985 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 59 16 11390 11374 1.50594 2.13281 0.0571799 0.0829573 2024-09-22T03:16:20.095 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:16:20.085+0000 7ffa83a58640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:16:20.100 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:16:20.099+0000 7ffa83a58640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:16:20.331 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-22T03:16:20.447 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:16:20.444+0000 7fb46f997640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:16:20.452 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:16:20.450+0000 7fb46f997640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:16:20.944 INFO:teuthology.orchestra.run.smithi153.stderr:reweighted osd.1 to 1 (10000) 2024-09-22T03:16:20.963 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-09-22T03:16:20.985 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:2024-09-22T03:16:20.984983+0000 min lat: 0.00692898 max lat: 20.7098 avg lat: 0.0824132 2024-09-22T03:16:20.985 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-22T03:16:20.985 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 60 16 11659 11643 1.51587 2.10156 0.0613409 0.0824132 2024-09-22T03:16:21.099 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:16:21.087+0000 7fb9dc8d7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:16:21.103 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:16:21.101+0000 7fb9dc8d7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:16:21.949 INFO:teuthology.orchestra.run.smithi153.stderr:reweighted osd.2 to 1 (10000) 2024-09-22T03:16:21.970 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-09-22T03:16:21.986 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 61 16 11933 11917 1.52611 2.14062 0.148767 0.0818604 2024-09-22T03:16:22.127 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:16:22.118+0000 7fdb8a197640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:16:22.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:16:22.130+0000 7fdb8a197640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:16:22.619 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-22T03:16:22.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:16:22.750+0000 7fbd3aa6c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:16:22.758 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:16:22.757+0000 7fbd3aa6c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:16:22.957 INFO:teuthology.orchestra.run.smithi153.stderr:reweighted osd.3 to 1 (10000) 2024-09-22T03:16:22.973 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-09-22T03:16:22.973 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-22T03:16:22.985 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 62 16 12201 12185 1.53526 2.09375 0.130138 0.0813145 2024-09-22T03:16:23.098 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:16:23.091+0000 7f2e5d597640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:16:23.102 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:16:23.101+0000 7f2e5d597640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:16:23.349 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-22T03:16:23.349 INFO:teuthology.orchestra.run.smithi153.stderr:dumped all 2024-09-22T03:16:23.363 INFO:teuthology.orchestra.run.smithi153.stdout:{"pg_ready":true,"pg_map":{"version":349,"stamp":"2024-09-22T03:16:22.956414+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":94700083,"num_objects":1440,"num_object_clones":0,"num_object_copies":4320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":11627,"num_write_kb":93800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":322,"num_bytes_recovered":21020672,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":11578,"ondisk_log_size":11578,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":299312,"kb_used_data":130216,"kb_used_omap":0,"kb_used_meta":169024,"kb_avail":377188048,"statfs":{"total":386547056640,"available":386240561152,"internally_reserved":0,"allocated":133341184,"data_stored":282225504,"data_compressed":14410466,"data_compressed_allocated":129818624,"data_compressed_original":280797184,"omap_allocated":0,"internal_metadata":173080576},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":6488064,"num_objects":99,"num_object_clones":0,"num_object_copies":297,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":792,"num_write_kb":6336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.045053"},"pg_stats":[{"pgid":"7.f","version":"101'760","reported_seq":829,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-22T03:16:18.964767+0000","last_change":"2024-09-22T03:15:33.972925+0000","last_active":"2024-09-22T03:16:18.964767+0000","last_peered":"2024-09-22T03:16:18.964767+0000","last_clean":"2024-09-22T03:16:18.964767+0000","last_became_active":"2024-09-22T03:15:33.967199+0000","last_became_peered":"2024-09-22T03:15:33.967199+0000","last_unstale":"2024-09-22T03:16:18.964767+0000","last_undegraded":"2024-09-22T03:16:18.964767+0000","last_fullsized":"2024-09-22T03:16:18.964767+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_clean_scrub_stamp":"2024-09-22T03:15:18.897706+0000","objects_scrubbed":0,"log_size":760,"log_dups_size":0,"ondisk_log_size":760,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6225920,"num_objects":95,"num_object_clones":0,"num_object_copies":285,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":760,"num_write_kb":6080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":1245184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.e","version":"102'711","reported_seq":773,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-22T03:16:19.149911+0000","last_change":"2024-09-22T03:16:18.028856+0000","last_active":"2024-09-22T03:16:19.149911+0000","last_peered":"2024-09-22T03:16:19.149911+0000","last_clean":"2024-09-22T03:16:19.149911+0000","last_became_active":"2024-09-22T03:16:18.021495+0000","last_became_peered":"2024-09-22T03:16:18.021495+0000","last_unstale":"2024-09-22T03:16:19.149911+0000","last_undegraded":"2024-09-22T03:16:19.149911+0000","last_fullsized":"2024-09-22T03:16:19.149911+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_clean_scrub_stamp":"2024-09-22T03:15:18.897706+0000","objects_scrubbed":0,"log_size":711,"log_dups_size":0,"ondisk_log_size":711,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5832704,"num_objects":89,"num_object_clones":0,"num_object_copies":267,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":712,"num_write_kb":5696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.d","version":"103'768","reported_seq":891,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-22T03:16:20.959408+0000","last_change":"2024-09-22T03:16:18.042408+0000","last_active":"2024-09-22T03:16:20.959408+0000","last_peered":"2024-09-22T03:16:20.959408+0000","last_clean":"2024-09-22T03:16:20.959408+0000","last_became_active":"2024-09-22T03:16:18.029899+0000","last_became_peered":"2024-09-22T03:16:18.029899+0000","last_unstale":"2024-09-22T03:16:20.959408+0000","last_undegraded":"2024-09-22T03:16:20.959408+0000","last_fullsized":"2024-09-22T03:16:20.959408+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_clean_scrub_stamp":"2024-09-22T03:15:18.897706+0000","objects_scrubbed":0,"log_size":768,"log_dups_size":0,"ondisk_log_size":768,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6291456,"num_objects":96,"num_object_clones":0,"num_object_copies":288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":768,"num_write_kb":6144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":1835008,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.c","version":"104'831","reported_seq":931,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-22T03:16:21.363209+0000","last_change":"2024-09-22T03:16:18.031863+0000","last_active":"2024-09-22T03:16:21.363209+0000","last_peered":"2024-09-22T03:16:21.363209+0000","last_clean":"2024-09-22T03:16:21.363209+0000","last_became_active":"2024-09-22T03:16:18.022365+0000","last_became_peered":"2024-09-22T03:16:18.022365+0000","last_unstale":"2024-09-22T03:16:21.363209+0000","last_undegraded":"2024-09-22T03:16:21.363209+0000","last_fullsized":"2024-09-22T03:16:21.363209+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_clean_scrub_stamp":"2024-09-22T03:15:18.897706+0000","objects_scrubbed":0,"log_size":831,"log_dups_size":0,"ondisk_log_size":831,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6815744,"num_objects":104,"num_object_clones":0,"num_object_copies":312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":832,"num_write_kb":6656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":1048576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.b","version":"98'696","reported_seq":744,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-22T03:16:18.966326+0000","last_change":"2024-09-22T03:15:34.925809+0000","last_active":"2024-09-22T03:16:18.966326+0000","last_peered":"2024-09-22T03:16:18.966326+0000","last_clean":"2024-09-22T03:16:18.966326+0000","last_became_active":"2024-09-22T03:15:34.920855+0000","last_became_peered":"2024-09-22T03:15:34.920855+0000","last_unstale":"2024-09-22T03:16:18.966326+0000","last_undegraded":"2024-09-22T03:16:18.966326+0000","last_fullsized":"2024-09-22T03:16:18.966326+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_clean_scrub_stamp":"2024-09-22T03:15:18.897706+0000","objects_scrubbed":0,"log_size":696,"log_dups_size":0,"ondisk_log_size":696,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5701632,"num_objects":87,"num_object_clones":0,"num_object_copies":261,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":696,"num_write_kb":5568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":2031616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"101'640","reported_seq":692,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-22T03:16:18.967053+0000","last_change":"2024-09-22T03:16:18.015447+0000","last_active":"2024-09-22T03:16:18.967053+0000","last_peered":"2024-09-22T03:16:18.967053+0000","last_clean":"2024-09-22T03:16:18.967053+0000","last_became_active":"2024-09-22T03:16:18.012158+0000","last_became_peered":"2024-09-22T03:16:18.012158+0000","last_unstale":"2024-09-22T03:16:18.967053+0000","last_undegraded":"2024-09-22T03:16:18.967053+0000","last_fullsized":"2024-09-22T03:16:18.967053+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_clean_scrub_stamp":"2024-09-22T03:15:18.897706+0000","objects_scrubbed":0,"log_size":640,"log_dups_size":0,"ondisk_log_size":640,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5242880,"num_objects":80,"num_object_clones":0,"num_object_copies":240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":640,"num_write_kb":5120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":1507328,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.9","version":"103'712","reported_seq":815,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-22T03:16:20.960725+0000","last_change":"2024-09-22T03:16:18.095024+0000","last_active":"2024-09-22T03:16:20.960725+0000","last_peered":"2024-09-22T03:16:20.960725+0000","last_clean":"2024-09-22T03:16:20.960725+0000","last_became_active":"2024-09-22T03:16:18.088681+0000","last_became_peered":"2024-09-22T03:16:18.088681+0000","last_unstale":"2024-09-22T03:16:20.960725+0000","last_undegraded":"2024-09-22T03:16:20.960725+0000","last_fullsized":"2024-09-22T03:16:20.960725+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_clean_scrub_stamp":"2024-09-22T03:15:18.897706+0000","objects_scrubbed":0,"log_size":712,"log_dups_size":0,"ondisk_log_size":712,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5832704,"num_objects":89,"num_object_clones":0,"num_object_copies":267,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":712,"num_write_kb":5696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":1179648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.8","version":"101'680","reported_seq":787,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-22T03:16:18.925741+0000","last_change":"2024-09-22T03:15:33.946927+0000","last_active":"2024-09-22T03:16:18.925741+0000","last_peered":"2024-09-22T03:16:18.925741+0000","last_clean":"2024-09-22T03:16:18.925741+0000","last_became_active":"2024-09-22T03:15:33.940267+0000","last_became_peered":"2024-09-22T03:15:33.940267+0000","last_unstale":"2024-09-22T03:16:18.925741+0000","last_undegraded":"2024-09-22T03:16:18.925741+0000","last_fullsized":"2024-09-22T03:16:18.925741+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_clean_scrub_stamp":"2024-09-22T03:15:18.897706+0000","objects_scrubbed":0,"log_size":680,"log_dups_size":0,"ondisk_log_size":680,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5570560,"num_objects":85,"num_object_clones":0,"num_object_copies":255,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":680,"num_write_kb":5440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":1507328,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.6","version":"102'775","reported_seq":856,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-22T03:16:19.102463+0000","last_change":"2024-09-22T03:15:32.999617+0000","last_active":"2024-09-22T03:16:19.102463+0000","last_peered":"2024-09-22T03:16:19.102463+0000","last_clean":"2024-09-22T03:16:19.102463+0000","last_became_active":"2024-09-22T03:15:32.993508+0000","last_became_peered":"2024-09-22T03:15:32.993508+0000","last_unstale":"2024-09-22T03:16:19.102463+0000","last_undegraded":"2024-09-22T03:16:19.102463+0000","last_fullsized":"2024-09-22T03:16:19.102463+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_clean_scrub_stamp":"2024-09-22T03:15:18.897706+0000","objects_scrubbed":0,"log_size":775,"log_dups_size":0,"ondisk_log_size":775,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6356992,"num_objects":97,"num_object_clones":0,"num_object_copies":291,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":776,"num_write_kb":6208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":491520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":144,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-22T03:16:18.967721+0000","last_change":"2024-09-22T03:16:18.969989+0000","last_active":"2024-09-22T03:16:18.967721+0000","last_peered":"2024-09-22T03:16:18.967721+0000","last_clean":"2024-09-22T03:16:18.967721+0000","last_became_active":"2024-09-22T03:16:18.003684+0000","last_became_peered":"2024-09-22T03:16:18.003684+0000","last_unstale":"2024-09-22T03:16:18.967721+0000","last_undegraded":"2024-09-22T03:16:18.967721+0000","last_fullsized":"2024-09-22T03:16:18.967721+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.7","version":"102'831","reported_seq":949,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-22T03:16:19.070502+0000","last_change":"2024-09-22T03:15:33.949109+0000","last_active":"2024-09-22T03:16:19.070502+0000","last_peered":"2024-09-22T03:16:19.070502+0000","last_clean":"2024-09-22T03:16:19.070502+0000","last_became_active":"2024-09-22T03:15:33.943481+0000","last_became_peered":"2024-09-22T03:15:33.943481+0000","last_unstale":"2024-09-22T03:16:19.070502+0000","last_undegraded":"2024-09-22T03:16:19.070502+0000","last_fullsized":"2024-09-22T03:16:19.070502+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_clean_scrub_stamp":"2024-09-22T03:15:18.897706+0000","objects_scrubbed":0,"log_size":831,"log_dups_size":0,"ondisk_log_size":831,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6815744,"num_objects":104,"num_object_clones":0,"num_object_copies":312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":832,"num_write_kb":6656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":1835008,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":233,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-22T03:16:20.953887+0000","last_change":"2024-09-22T03:16:18.925677+0000","last_active":"2024-09-22T03:16:20.953887+0000","last_peered":"2024-09-22T03:16:20.953887+0000","last_clean":"2024-09-22T03:16:20.953887+0000","last_became_active":"2024-09-22T03:16:17.986660+0000","last_became_peered":"2024-09-22T03:16:17.986660+0000","last_unstale":"2024-09-22T03:16:20.953887+0000","last_undegraded":"2024-09-22T03:16:20.953887+0000","last_fullsized":"2024-09-22T03:16:20.953887+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.4","version":"104'767","reported_seq":877,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-22T03:16:21.576830+0000","last_change":"2024-09-22T03:16:18.072690+0000","last_active":"2024-09-22T03:16:21.576830+0000","last_peered":"2024-09-22T03:16:21.576830+0000","last_clean":"2024-09-22T03:16:21.576830+0000","last_became_active":"2024-09-22T03:16:18.063063+0000","last_became_peered":"2024-09-22T03:16:18.063063+0000","last_unstale":"2024-09-22T03:16:21.576830+0000","last_undegraded":"2024-09-22T03:16:21.576830+0000","last_fullsized":"2024-09-22T03:16:21.576830+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_clean_scrub_stamp":"2024-09-22T03:15:18.897706+0000","objects_scrubbed":0,"log_size":767,"log_dups_size":0,"ondisk_log_size":767,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6291456,"num_objects":96,"num_object_clones":0,"num_object_copies":288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":768,"num_write_kb":6144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1376256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":230,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-22T03:16:20.961623+0000","last_change":"2024-09-22T03:16:18.926944+0000","last_active":"2024-09-22T03:16:20.961623+0000","last_peered":"2024-09-22T03:16:20.961623+0000","last_clean":"2024-09-22T03:16:20.961623+0000","last_became_active":"2024-09-22T03:16:18.083437+0000","last_became_peered":"2024-09-22T03:16:18.083437+0000","last_unstale":"2024-09-22T03:16:20.961623+0000","last_undegraded":"2024-09-22T03:16:20.961623+0000","last_fullsized":"2024-09-22T03:16:20.961623+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.5","version":"102'663","reported_seq":787,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-22T03:16:18.958453+0000","last_change":"2024-09-22T03:16:18.004962+0000","last_active":"2024-09-22T03:16:18.958453+0000","last_peered":"2024-09-22T03:16:18.958453+0000","last_clean":"2024-09-22T03:16:18.958453+0000","last_became_active":"2024-09-22T03:16:18.000028+0000","last_became_peered":"2024-09-22T03:16:18.000028+0000","last_unstale":"2024-09-22T03:16:18.958453+0000","last_undegraded":"2024-09-22T03:16:18.958453+0000","last_fullsized":"2024-09-22T03:16:18.958453+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_clean_scrub_stamp":"2024-09-22T03:15:18.897706+0000","objects_scrubbed":0,"log_size":663,"log_dups_size":0,"ondisk_log_size":663,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5439488,"num_objects":83,"num_object_clones":0,"num_object_copies":249,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":664,"num_write_kb":5312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":1966080,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":203,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-22T03:16:18.963261+0000","last_change":"2024-09-22T03:15:30.924363+0000","last_active":"2024-09-22T03:16:18.963261+0000","last_peered":"2024-09-22T03:16:18.963261+0000","last_clean":"2024-09-22T03:16:18.963261+0000","last_became_active":"2024-09-22T03:15:29.958549+0000","last_became_peered":"2024-09-22T03:15:29.958549+0000","last_unstale":"2024-09-22T03:16:18.963261+0000","last_undegraded":"2024-09-22T03:16:18.963261+0000","last_fullsized":"2024-09-22T03:16:18.963261+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"12'97","reported_seq":355,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-22T03:16:20.956259+0000","last_change":"2024-09-22T03:16:17.998607+0000","last_active":"2024-09-22T03:16:20.956259+0000","last_peered":"2024-09-22T03:16:20.956259+0000","last_clean":"2024-09-22T03:16:20.956259+0000","last_became_active":"2024-09-22T03:16:17.995046+0000","last_became_peered":"2024-09-22T03:16:17.995046+0000","last_unstale":"2024-09-22T03:16:20.956259+0000","last_undegraded":"2024-09-22T03:16:20.956259+0000","last_fullsized":"2024-09-22T03:16:20.956259+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.2","version":"102'679","reported_seq":755,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-22T03:16:19.056664+0000","last_change":"2024-09-22T03:15:33.953595+0000","last_active":"2024-09-22T03:16:19.056664+0000","last_peered":"2024-09-22T03:16:19.056664+0000","last_clean":"2024-09-22T03:16:19.056664+0000","last_became_active":"2024-09-22T03:15:33.947435+0000","last_became_peered":"2024-09-22T03:15:33.947435+0000","last_unstale":"2024-09-22T03:16:19.056664+0000","last_undegraded":"2024-09-22T03:16:19.056664+0000","last_fullsized":"2024-09-22T03:16:19.056664+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_clean_scrub_stamp":"2024-09-22T03:15:18.897706+0000","objects_scrubbed":0,"log_size":679,"log_dups_size":0,"ondisk_log_size":679,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5570560,"num_objects":85,"num_object_clones":0,"num_object_copies":255,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":680,"num_write_kb":5440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1376256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":232,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-22T03:16:18.924187+0000","last_change":"2024-09-22T03:16:18.925553+0000","last_active":"2024-09-22T03:16:18.924187+0000","last_peered":"2024-09-22T03:16:18.924187+0000","last_clean":"2024-09-22T03:16:18.924187+0000","last_became_active":"2024-09-22T03:16:18.021400+0000","last_became_peered":"2024-09-22T03:16:18.021400+0000","last_unstale":"2024-09-22T03:16:18.924187+0000","last_undegraded":"2024-09-22T03:16:18.924187+0000","last_fullsized":"2024-09-22T03:16:18.924187+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.3","version":"102'695","reported_seq":768,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-22T03:16:19.000126+0000","last_change":"2024-09-22T03:16:18.004775+0000","last_active":"2024-09-22T03:16:19.000126+0000","last_peered":"2024-09-22T03:16:19.000126+0000","last_clean":"2024-09-22T03:16:19.000126+0000","last_became_active":"2024-09-22T03:16:17.999612+0000","last_became_peered":"2024-09-22T03:16:17.999612+0000","last_unstale":"2024-09-22T03:16:19.000126+0000","last_undegraded":"2024-09-22T03:16:19.000126+0000","last_fullsized":"2024-09-22T03:16:19.000126+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_clean_scrub_stamp":"2024-09-22T03:15:18.897706+0000","objects_scrubbed":0,"log_size":695,"log_dups_size":0,"ondisk_log_size":695,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5701632,"num_objects":87,"num_object_clones":0,"num_object_copies":261,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":696,"num_write_kb":5568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1376256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":209,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-22T03:16:20.960510+0000","last_change":"2024-09-22T03:16:18.927373+0000","last_active":"2024-09-22T03:16:20.960510+0000","last_peered":"2024-09-22T03:16:20.960510+0000","last_clean":"2024-09-22T03:16:20.960510+0000","last_became_active":"2024-09-22T03:16:18.086190+0000","last_became_peered":"2024-09-22T03:16:18.086190+0000","last_unstale":"2024-09-22T03:16:20.960510+0000","last_undegraded":"2024-09-22T03:16:20.960510+0000","last_fullsized":"2024-09-22T03:16:20.960510+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.0","version":"104'639","reported_seq":752,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-22T03:16:21.632169+0000","last_change":"2024-09-22T03:16:18.066854+0000","last_active":"2024-09-22T03:16:21.632169+0000","last_peered":"2024-09-22T03:16:21.632169+0000","last_clean":"2024-09-22T03:16:21.632169+0000","last_became_active":"2024-09-22T03:16:18.055858+0000","last_became_peered":"2024-09-22T03:16:18.055858+0000","last_unstale":"2024-09-22T03:16:21.632169+0000","last_undegraded":"2024-09-22T03:16:21.632169+0000","last_fullsized":"2024-09-22T03:16:21.632169+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_clean_scrub_stamp":"2024-09-22T03:15:18.897706+0000","objects_scrubbed":0,"log_size":639,"log_dups_size":0,"ondisk_log_size":639,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5242880,"num_objects":80,"num_object_clones":0,"num_object_copies":240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":640,"num_write_kb":5120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1327104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":246,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-22T03:16:18.926407+0000","last_change":"2024-09-22T03:16:18.928041+0000","last_active":"2024-09-22T03:16:18.926407+0000","last_peered":"2024-09-22T03:16:18.926407+0000","last_clean":"2024-09-22T03:16:18.926407+0000","last_became_active":"2024-09-22T03:16:18.025488+0000","last_became_peered":"2024-09-22T03:16:18.025488+0000","last_unstale":"2024-09-22T03:16:18.926407+0000","last_undegraded":"2024-09-22T03:16:18.926407+0000","last_fullsized":"2024-09-22T03:16:18.926407+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1","version":"104'631","reported_seq":727,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-22T03:16:21.869553+0000","last_change":"2024-09-22T03:15:33.950769+0000","last_active":"2024-09-22T03:16:21.869553+0000","last_peered":"2024-09-22T03:16:21.869553+0000","last_clean":"2024-09-22T03:16:21.869553+0000","last_became_active":"2024-09-22T03:15:33.945111+0000","last_became_peered":"2024-09-22T03:15:33.945111+0000","last_unstale":"2024-09-22T03:16:21.869553+0000","last_undegraded":"2024-09-22T03:16:21.869553+0000","last_fullsized":"2024-09-22T03:16:21.869553+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_clean_scrub_stamp":"2024-09-22T03:15:18.897706+0000","objects_scrubbed":0,"log_size":631,"log_dups_size":0,"ondisk_log_size":631,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5177344,"num_objects":79,"num_object_clones":0,"num_object_copies":237,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":632,"num_write_kb":5056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":917504,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":239,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-22T03:16:18.924675+0000","last_change":"2024-09-22T03:15:30.958189+0000","last_active":"2024-09-22T03:16:18.924675+0000","last_peered":"2024-09-22T03:16:18.924675+0000","last_clean":"2024-09-22T03:16:18.924675+0000","last_became_active":"2024-09-22T03:15:29.980027+0000","last_became_peered":"2024-09-22T03:15:29.980027+0000","last_unstale":"2024-09-22T03:16:18.924675+0000","last_undegraded":"2024-09-22T03:16:18.924675+0000","last_fullsized":"2024-09-22T03:16:18.924675+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":94109696,"num_objects":1436,"num_object_clones":0,"num_object_copies":4308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11488,"num_write_kb":91904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":322,"num_bytes_recovered":21020672,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":130994176,"data_stored":281673728,"data_compressed":14454670,"data_compressed_allocated":130994176,"data_compressed_original":281673728,"omap_allocated":0,"internal_metadata":0},"log_size":11478,"ondisk_log_size":11478,"up":48,"acting":48,"num_store_stats":4},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1180736,"data_compressed":59924,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":87,"seq":373662154802,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":66124,"kb_used_data":32828,"kb_used_omap":0,"kb_used_meta":33280,"kb_avail":94305716,"statfs":{"total":96636764160,"available":96569053184,"internally_reserved":0,"allocated":33615872,"data_stored":85505966,"data_compressed":4310207,"data_compressed_allocated":32755712,"data_compressed_original":85164032,"omap_allocated":0,"internal_metadata":34078720},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":100,"seq":429496729604,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":67704,"kb_used_data":26788,"kb_used_omap":0,"kb_used_meta":40896,"kb_avail":94304136,"statfs":{"total":96636764160,"available":96567435264,"internally_reserved":0,"allocated":27430912,"data_stored":53937815,"data_compressed":2766133,"data_compressed_allocated":26542080,"data_compressed_original":53575680,"omap_allocated":0,"internal_metadata":41877504},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":53,"seq":227633266912,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":91716,"kb_used_data":37300,"kb_used_omap":0,"kb_used_meta":54400,"kb_avail":94280124,"statfs":{"total":96636764160,"available":96542846976,"internally_reserved":0,"allocated":38195200,"data_stored":75487694,"data_compressed":3878781,"data_compressed_allocated":37314560,"data_compressed_original":75128832,"omap_allocated":0,"internal_metadata":55705600},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":73,"seq":313532612713,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":73768,"kb_used_data":33300,"kb_used_omap":0,"kb_used_meta":40448,"kb_avail":94298072,"statfs":{"total":96636764160,"available":96561225728,"internally_reserved":0,"allocated":34099200,"data_stored":67294029,"data_compressed":3455345,"data_compressed_allocated":33206272,"data_compressed_original":66928640,"omap_allocated":0,"internal_metadata":41418752},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29962,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29962,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":33705984,"data_stored":67436544,"data_compressed":3481219,"data_compressed_allocated":33705984,"data_compressed_original":67436544,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":37457920,"data_stored":74924032,"data_compressed":3867854,"data_compressed_allocated":37457920,"data_compressed_original":74924032,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":26869760,"data_stored":53739520,"data_compressed":2774241,"data_compressed_allocated":26869760,"data_compressed_original":53739520,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":32960512,"data_stored":85573632,"data_compressed":4331356,"data_compressed_allocated":32960512,"data_compressed_original":85573632,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-22T03:16:23.364 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-22T03:16:23.504 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:16:23.498+0000 7f6e73597640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:16:23.507 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:16:23.506+0000 7f6e73597640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:16:23.763 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-22T03:16:23.763 INFO:teuthology.orchestra.run.smithi153.stderr:dumped all 2024-09-22T03:16:23.783 INFO:teuthology.orchestra.run.smithi153.stdout:{"pg_ready":true,"pg_map":{"version":349,"stamp":"2024-09-22T03:16:22.956414+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":94700083,"num_objects":1440,"num_object_clones":0,"num_object_copies":4320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":11627,"num_write_kb":93800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":322,"num_bytes_recovered":21020672,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":11578,"ondisk_log_size":11578,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":299312,"kb_used_data":130216,"kb_used_omap":0,"kb_used_meta":169024,"kb_avail":377188048,"statfs":{"total":386547056640,"available":386240561152,"internally_reserved":0,"allocated":133341184,"data_stored":282225504,"data_compressed":14410466,"data_compressed_allocated":129818624,"data_compressed_original":280797184,"omap_allocated":0,"internal_metadata":173080576},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":6488064,"num_objects":99,"num_object_clones":0,"num_object_copies":297,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":792,"num_write_kb":6336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.045053"},"pg_stats":[{"pgid":"7.f","version":"101'760","reported_seq":829,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-22T03:16:18.964767+0000","last_change":"2024-09-22T03:15:33.972925+0000","last_active":"2024-09-22T03:16:18.964767+0000","last_peered":"2024-09-22T03:16:18.964767+0000","last_clean":"2024-09-22T03:16:18.964767+0000","last_became_active":"2024-09-22T03:15:33.967199+0000","last_became_peered":"2024-09-22T03:15:33.967199+0000","last_unstale":"2024-09-22T03:16:18.964767+0000","last_undegraded":"2024-09-22T03:16:18.964767+0000","last_fullsized":"2024-09-22T03:16:18.964767+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_clean_scrub_stamp":"2024-09-22T03:15:18.897706+0000","objects_scrubbed":0,"log_size":760,"log_dups_size":0,"ondisk_log_size":760,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6225920,"num_objects":95,"num_object_clones":0,"num_object_copies":285,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":760,"num_write_kb":6080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":1245184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.e","version":"102'711","reported_seq":773,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-22T03:16:19.149911+0000","last_change":"2024-09-22T03:16:18.028856+0000","last_active":"2024-09-22T03:16:19.149911+0000","last_peered":"2024-09-22T03:16:19.149911+0000","last_clean":"2024-09-22T03:16:19.149911+0000","last_became_active":"2024-09-22T03:16:18.021495+0000","last_became_peered":"2024-09-22T03:16:18.021495+0000","last_unstale":"2024-09-22T03:16:19.149911+0000","last_undegraded":"2024-09-22T03:16:19.149911+0000","last_fullsized":"2024-09-22T03:16:19.149911+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_clean_scrub_stamp":"2024-09-22T03:15:18.897706+0000","objects_scrubbed":0,"log_size":711,"log_dups_size":0,"ondisk_log_size":711,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5832704,"num_objects":89,"num_object_clones":0,"num_object_copies":267,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":712,"num_write_kb":5696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.d","version":"103'768","reported_seq":891,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-22T03:16:20.959408+0000","last_change":"2024-09-22T03:16:18.042408+0000","last_active":"2024-09-22T03:16:20.959408+0000","last_peered":"2024-09-22T03:16:20.959408+0000","last_clean":"2024-09-22T03:16:20.959408+0000","last_became_active":"2024-09-22T03:16:18.029899+0000","last_became_peered":"2024-09-22T03:16:18.029899+0000","last_unstale":"2024-09-22T03:16:20.959408+0000","last_undegraded":"2024-09-22T03:16:20.959408+0000","last_fullsized":"2024-09-22T03:16:20.959408+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_clean_scrub_stamp":"2024-09-22T03:15:18.897706+0000","objects_scrubbed":0,"log_size":768,"log_dups_size":0,"ondisk_log_size":768,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6291456,"num_objects":96,"num_object_clones":0,"num_object_copies":288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":768,"num_write_kb":6144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":1835008,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.c","version":"104'831","reported_seq":931,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-22T03:16:21.363209+0000","last_change":"2024-09-22T03:16:18.031863+0000","last_active":"2024-09-22T03:16:21.363209+0000","last_peered":"2024-09-22T03:16:21.363209+0000","last_clean":"2024-09-22T03:16:21.363209+0000","last_became_active":"2024-09-22T03:16:18.022365+0000","last_became_peered":"2024-09-22T03:16:18.022365+0000","last_unstale":"2024-09-22T03:16:21.363209+0000","last_undegraded":"2024-09-22T03:16:21.363209+0000","last_fullsized":"2024-09-22T03:16:21.363209+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_clean_scrub_stamp":"2024-09-22T03:15:18.897706+0000","objects_scrubbed":0,"log_size":831,"log_dups_size":0,"ondisk_log_size":831,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6815744,"num_objects":104,"num_object_clones":0,"num_object_copies":312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":832,"num_write_kb":6656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":1048576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.b","version":"98'696","reported_seq":744,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-22T03:16:18.966326+0000","last_change":"2024-09-22T03:15:34.925809+0000","last_active":"2024-09-22T03:16:18.966326+0000","last_peered":"2024-09-22T03:16:18.966326+0000","last_clean":"2024-09-22T03:16:18.966326+0000","last_became_active":"2024-09-22T03:15:34.920855+0000","last_became_peered":"2024-09-22T03:15:34.920855+0000","last_unstale":"2024-09-22T03:16:18.966326+0000","last_undegraded":"2024-09-22T03:16:18.966326+0000","last_fullsized":"2024-09-22T03:16:18.966326+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_clean_scrub_stamp":"2024-09-22T03:15:18.897706+0000","objects_scrubbed":0,"log_size":696,"log_dups_size":0,"ondisk_log_size":696,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5701632,"num_objects":87,"num_object_clones":0,"num_object_copies":261,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":696,"num_write_kb":5568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":2031616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"101'640","reported_seq":692,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-22T03:16:18.967053+0000","last_change":"2024-09-22T03:16:18.015447+0000","last_active":"2024-09-22T03:16:18.967053+0000","last_peered":"2024-09-22T03:16:18.967053+0000","last_clean":"2024-09-22T03:16:18.967053+0000","last_became_active":"2024-09-22T03:16:18.012158+0000","last_became_peered":"2024-09-22T03:16:18.012158+0000","last_unstale":"2024-09-22T03:16:18.967053+0000","last_undegraded":"2024-09-22T03:16:18.967053+0000","last_fullsized":"2024-09-22T03:16:18.967053+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_clean_scrub_stamp":"2024-09-22T03:15:18.897706+0000","objects_scrubbed":0,"log_size":640,"log_dups_size":0,"ondisk_log_size":640,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5242880,"num_objects":80,"num_object_clones":0,"num_object_copies":240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":640,"num_write_kb":5120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":1507328,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.9","version":"103'712","reported_seq":815,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-22T03:16:20.960725+0000","last_change":"2024-09-22T03:16:18.095024+0000","last_active":"2024-09-22T03:16:20.960725+0000","last_peered":"2024-09-22T03:16:20.960725+0000","last_clean":"2024-09-22T03:16:20.960725+0000","last_became_active":"2024-09-22T03:16:18.088681+0000","last_became_peered":"2024-09-22T03:16:18.088681+0000","last_unstale":"2024-09-22T03:16:20.960725+0000","last_undegraded":"2024-09-22T03:16:20.960725+0000","last_fullsized":"2024-09-22T03:16:20.960725+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_clean_scrub_stamp":"2024-09-22T03:15:18.897706+0000","objects_scrubbed":0,"log_size":712,"log_dups_size":0,"ondisk_log_size":712,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5832704,"num_objects":89,"num_object_clones":0,"num_object_copies":267,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":712,"num_write_kb":5696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":1179648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.8","version":"101'680","reported_seq":787,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-22T03:16:18.925741+0000","last_change":"2024-09-22T03:15:33.946927+0000","last_active":"2024-09-22T03:16:18.925741+0000","last_peered":"2024-09-22T03:16:18.925741+0000","last_clean":"2024-09-22T03:16:18.925741+0000","last_became_active":"2024-09-22T03:15:33.940267+0000","last_became_peered":"2024-09-22T03:15:33.940267+0000","last_unstale":"2024-09-22T03:16:18.925741+0000","last_undegraded":"2024-09-22T03:16:18.925741+0000","last_fullsized":"2024-09-22T03:16:18.925741+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_clean_scrub_stamp":"2024-09-22T03:15:18.897706+0000","objects_scrubbed":0,"log_size":680,"log_dups_size":0,"ondisk_log_size":680,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5570560,"num_objects":85,"num_object_clones":0,"num_object_copies":255,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":680,"num_write_kb":5440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":1507328,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.6","version":"102'775","reported_seq":856,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-22T03:16:19.102463+0000","last_change":"2024-09-22T03:15:32.999617+0000","last_active":"2024-09-22T03:16:19.102463+0000","last_peered":"2024-09-22T03:16:19.102463+0000","last_clean":"2024-09-22T03:16:19.102463+0000","last_became_active":"2024-09-22T03:15:32.993508+0000","last_became_peered":"2024-09-22T03:15:32.993508+0000","last_unstale":"2024-09-22T03:16:19.102463+0000","last_undegraded":"2024-09-22T03:16:19.102463+0000","last_fullsized":"2024-09-22T03:16:19.102463+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_clean_scrub_stamp":"2024-09-22T03:15:18.897706+0000","objects_scrubbed":0,"log_size":775,"log_dups_size":0,"ondisk_log_size":775,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6356992,"num_objects":97,"num_object_clones":0,"num_object_copies":291,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":776,"num_write_kb":6208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":491520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":144,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-22T03:16:18.967721+0000","last_change":"2024-09-22T03:16:18.969989+0000","last_active":"2024-09-22T03:16:18.967721+0000","last_peered":"2024-09-22T03:16:18.967721+0000","last_clean":"2024-09-22T03:16:18.967721+0000","last_became_active":"2024-09-22T03:16:18.003684+0000","last_became_peered":"2024-09-22T03:16:18.003684+0000","last_unstale":"2024-09-22T03:16:18.967721+0000","last_undegraded":"2024-09-22T03:16:18.967721+0000","last_fullsized":"2024-09-22T03:16:18.967721+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.7","version":"102'831","reported_seq":949,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-22T03:16:19.070502+0000","last_change":"2024-09-22T03:15:33.949109+0000","last_active":"2024-09-22T03:16:19.070502+0000","last_peered":"2024-09-22T03:16:19.070502+0000","last_clean":"2024-09-22T03:16:19.070502+0000","last_became_active":"2024-09-22T03:15:33.943481+0000","last_became_peered":"2024-09-22T03:15:33.943481+0000","last_unstale":"2024-09-22T03:16:19.070502+0000","last_undegraded":"2024-09-22T03:16:19.070502+0000","last_fullsized":"2024-09-22T03:16:19.070502+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_clean_scrub_stamp":"2024-09-22T03:15:18.897706+0000","objects_scrubbed":0,"log_size":831,"log_dups_size":0,"ondisk_log_size":831,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6815744,"num_objects":104,"num_object_clones":0,"num_object_copies":312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":832,"num_write_kb":6656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":1835008,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":233,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-22T03:16:20.953887+0000","last_change":"2024-09-22T03:16:18.925677+0000","last_active":"2024-09-22T03:16:20.953887+0000","last_peered":"2024-09-22T03:16:20.953887+0000","last_clean":"2024-09-22T03:16:20.953887+0000","last_became_active":"2024-09-22T03:16:17.986660+0000","last_became_peered":"2024-09-22T03:16:17.986660+0000","last_unstale":"2024-09-22T03:16:20.953887+0000","last_undegraded":"2024-09-22T03:16:20.953887+0000","last_fullsized":"2024-09-22T03:16:20.953887+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.4","version":"104'767","reported_seq":877,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-22T03:16:21.576830+0000","last_change":"2024-09-22T03:16:18.072690+0000","last_active":"2024-09-22T03:16:21.576830+0000","last_peered":"2024-09-22T03:16:21.576830+0000","last_clean":"2024-09-22T03:16:21.576830+0000","last_became_active":"2024-09-22T03:16:18.063063+0000","last_became_peered":"2024-09-22T03:16:18.063063+0000","last_unstale":"2024-09-22T03:16:21.576830+0000","last_undegraded":"2024-09-22T03:16:21.576830+0000","last_fullsized":"2024-09-22T03:16:21.576830+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_clean_scrub_stamp":"2024-09-22T03:15:18.897706+0000","objects_scrubbed":0,"log_size":767,"log_dups_size":0,"ondisk_log_size":767,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6291456,"num_objects":96,"num_object_clones":0,"num_object_copies":288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":768,"num_write_kb":6144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1376256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":230,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-22T03:16:20.961623+0000","last_change":"2024-09-22T03:16:18.926944+0000","last_active":"2024-09-22T03:16:20.961623+0000","last_peered":"2024-09-22T03:16:20.961623+0000","last_clean":"2024-09-22T03:16:20.961623+0000","last_became_active":"2024-09-22T03:16:18.083437+0000","last_became_peered":"2024-09-22T03:16:18.083437+0000","last_unstale":"2024-09-22T03:16:20.961623+0000","last_undegraded":"2024-09-22T03:16:20.961623+0000","last_fullsized":"2024-09-22T03:16:20.961623+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.5","version":"102'663","reported_seq":787,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-22T03:16:18.958453+0000","last_change":"2024-09-22T03:16:18.004962+0000","last_active":"2024-09-22T03:16:18.958453+0000","last_peered":"2024-09-22T03:16:18.958453+0000","last_clean":"2024-09-22T03:16:18.958453+0000","last_became_active":"2024-09-22T03:16:18.000028+0000","last_became_peered":"2024-09-22T03:16:18.000028+0000","last_unstale":"2024-09-22T03:16:18.958453+0000","last_undegraded":"2024-09-22T03:16:18.958453+0000","last_fullsized":"2024-09-22T03:16:18.958453+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_clean_scrub_stamp":"2024-09-22T03:15:18.897706+0000","objects_scrubbed":0,"log_size":663,"log_dups_size":0,"ondisk_log_size":663,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5439488,"num_objects":83,"num_object_clones":0,"num_object_copies":249,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":664,"num_write_kb":5312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":1966080,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":203,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-22T03:16:18.963261+0000","last_change":"2024-09-22T03:15:30.924363+0000","last_active":"2024-09-22T03:16:18.963261+0000","last_peered":"2024-09-22T03:16:18.963261+0000","last_clean":"2024-09-22T03:16:18.963261+0000","last_became_active":"2024-09-22T03:15:29.958549+0000","last_became_peered":"2024-09-22T03:15:29.958549+0000","last_unstale":"2024-09-22T03:16:18.963261+0000","last_undegraded":"2024-09-22T03:16:18.963261+0000","last_fullsized":"2024-09-22T03:16:18.963261+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"12'97","reported_seq":355,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-22T03:16:20.956259+0000","last_change":"2024-09-22T03:16:17.998607+0000","last_active":"2024-09-22T03:16:20.956259+0000","last_peered":"2024-09-22T03:16:20.956259+0000","last_clean":"2024-09-22T03:16:20.956259+0000","last_became_active":"2024-09-22T03:16:17.995046+0000","last_became_peered":"2024-09-22T03:16:17.995046+0000","last_unstale":"2024-09-22T03:16:20.956259+0000","last_undegraded":"2024-09-22T03:16:20.956259+0000","last_fullsized":"2024-09-22T03:16:20.956259+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.2","version":"102'679","reported_seq":755,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-22T03:16:19.056664+0000","last_change":"2024-09-22T03:15:33.953595+0000","last_active":"2024-09-22T03:16:19.056664+0000","last_peered":"2024-09-22T03:16:19.056664+0000","last_clean":"2024-09-22T03:16:19.056664+0000","last_became_active":"2024-09-22T03:15:33.947435+0000","last_became_peered":"2024-09-22T03:15:33.947435+0000","last_unstale":"2024-09-22T03:16:19.056664+0000","last_undegraded":"2024-09-22T03:16:19.056664+0000","last_fullsized":"2024-09-22T03:16:19.056664+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_clean_scrub_stamp":"2024-09-22T03:15:18.897706+0000","objects_scrubbed":0,"log_size":679,"log_dups_size":0,"ondisk_log_size":679,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5570560,"num_objects":85,"num_object_clones":0,"num_object_copies":255,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":680,"num_write_kb":5440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1376256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":232,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-22T03:16:18.924187+0000","last_change":"2024-09-22T03:16:18.925553+0000","last_active":"2024-09-22T03:16:18.924187+0000","last_peered":"2024-09-22T03:16:18.924187+0000","last_clean":"2024-09-22T03:16:18.924187+0000","last_became_active":"2024-09-22T03:16:18.021400+0000","last_became_peered":"2024-09-22T03:16:18.021400+0000","last_unstale":"2024-09-22T03:16:18.924187+0000","last_undegraded":"2024-09-22T03:16:18.924187+0000","last_fullsized":"2024-09-22T03:16:18.924187+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.3","version":"102'695","reported_seq":768,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-22T03:16:19.000126+0000","last_change":"2024-09-22T03:16:18.004775+0000","last_active":"2024-09-22T03:16:19.000126+0000","last_peered":"2024-09-22T03:16:19.000126+0000","last_clean":"2024-09-22T03:16:19.000126+0000","last_became_active":"2024-09-22T03:16:17.999612+0000","last_became_peered":"2024-09-22T03:16:17.999612+0000","last_unstale":"2024-09-22T03:16:19.000126+0000","last_undegraded":"2024-09-22T03:16:19.000126+0000","last_fullsized":"2024-09-22T03:16:19.000126+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_clean_scrub_stamp":"2024-09-22T03:15:18.897706+0000","objects_scrubbed":0,"log_size":695,"log_dups_size":0,"ondisk_log_size":695,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5701632,"num_objects":87,"num_object_clones":0,"num_object_copies":261,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":696,"num_write_kb":5568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1376256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":209,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-22T03:16:20.960510+0000","last_change":"2024-09-22T03:16:18.927373+0000","last_active":"2024-09-22T03:16:20.960510+0000","last_peered":"2024-09-22T03:16:20.960510+0000","last_clean":"2024-09-22T03:16:20.960510+0000","last_became_active":"2024-09-22T03:16:18.086190+0000","last_became_peered":"2024-09-22T03:16:18.086190+0000","last_unstale":"2024-09-22T03:16:20.960510+0000","last_undegraded":"2024-09-22T03:16:20.960510+0000","last_fullsized":"2024-09-22T03:16:20.960510+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.0","version":"104'639","reported_seq":752,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-22T03:16:21.632169+0000","last_change":"2024-09-22T03:16:18.066854+0000","last_active":"2024-09-22T03:16:21.632169+0000","last_peered":"2024-09-22T03:16:21.632169+0000","last_clean":"2024-09-22T03:16:21.632169+0000","last_became_active":"2024-09-22T03:16:18.055858+0000","last_became_peered":"2024-09-22T03:16:18.055858+0000","last_unstale":"2024-09-22T03:16:21.632169+0000","last_undegraded":"2024-09-22T03:16:21.632169+0000","last_fullsized":"2024-09-22T03:16:21.632169+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_clean_scrub_stamp":"2024-09-22T03:15:18.897706+0000","objects_scrubbed":0,"log_size":639,"log_dups_size":0,"ondisk_log_size":639,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5242880,"num_objects":80,"num_object_clones":0,"num_object_copies":240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":640,"num_write_kb":5120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1327104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":246,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-22T03:16:18.926407+0000","last_change":"2024-09-22T03:16:18.928041+0000","last_active":"2024-09-22T03:16:18.926407+0000","last_peered":"2024-09-22T03:16:18.926407+0000","last_clean":"2024-09-22T03:16:18.926407+0000","last_became_active":"2024-09-22T03:16:18.025488+0000","last_became_peered":"2024-09-22T03:16:18.025488+0000","last_unstale":"2024-09-22T03:16:18.926407+0000","last_undegraded":"2024-09-22T03:16:18.926407+0000","last_fullsized":"2024-09-22T03:16:18.926407+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1","version":"104'631","reported_seq":727,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-22T03:16:21.869553+0000","last_change":"2024-09-22T03:15:33.950769+0000","last_active":"2024-09-22T03:16:21.869553+0000","last_peered":"2024-09-22T03:16:21.869553+0000","last_clean":"2024-09-22T03:16:21.869553+0000","last_became_active":"2024-09-22T03:15:33.945111+0000","last_became_peered":"2024-09-22T03:15:33.945111+0000","last_unstale":"2024-09-22T03:16:21.869553+0000","last_undegraded":"2024-09-22T03:16:21.869553+0000","last_fullsized":"2024-09-22T03:16:21.869553+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_clean_scrub_stamp":"2024-09-22T03:15:18.897706+0000","objects_scrubbed":0,"log_size":631,"log_dups_size":0,"ondisk_log_size":631,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5177344,"num_objects":79,"num_object_clones":0,"num_object_copies":237,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":632,"num_write_kb":5056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":917504,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":239,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-22T03:16:18.924675+0000","last_change":"2024-09-22T03:15:30.958189+0000","last_active":"2024-09-22T03:16:18.924675+0000","last_peered":"2024-09-22T03:16:18.924675+0000","last_clean":"2024-09-22T03:16:18.924675+0000","last_became_active":"2024-09-22T03:15:29.980027+0000","last_became_peered":"2024-09-22T03:15:29.980027+0000","last_unstale":"2024-09-22T03:16:18.924675+0000","last_undegraded":"2024-09-22T03:16:18.924675+0000","last_fullsized":"2024-09-22T03:16:18.924675+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":94109696,"num_objects":1436,"num_object_clones":0,"num_object_copies":4308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11488,"num_write_kb":91904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":322,"num_bytes_recovered":21020672,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":130994176,"data_stored":281673728,"data_compressed":14454670,"data_compressed_allocated":130994176,"data_compressed_original":281673728,"omap_allocated":0,"internal_metadata":0},"log_size":11478,"ondisk_log_size":11478,"up":48,"acting":48,"num_store_stats":4},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1180736,"data_compressed":59924,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":87,"seq":373662154802,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":66124,"kb_used_data":32828,"kb_used_omap":0,"kb_used_meta":33280,"kb_avail":94305716,"statfs":{"total":96636764160,"available":96569053184,"internally_reserved":0,"allocated":33615872,"data_stored":85505966,"data_compressed":4310207,"data_compressed_allocated":32755712,"data_compressed_original":85164032,"omap_allocated":0,"internal_metadata":34078720},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":100,"seq":429496729604,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":67704,"kb_used_data":26788,"kb_used_omap":0,"kb_used_meta":40896,"kb_avail":94304136,"statfs":{"total":96636764160,"available":96567435264,"internally_reserved":0,"allocated":27430912,"data_stored":53937815,"data_compressed":2766133,"data_compressed_allocated":26542080,"data_compressed_original":53575680,"omap_allocated":0,"internal_metadata":41877504},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":53,"seq":227633266912,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":91716,"kb_used_data":37300,"kb_used_omap":0,"kb_used_meta":54400,"kb_avail":94280124,"statfs":{"total":96636764160,"available":96542846976,"internally_reserved":0,"allocated":38195200,"data_stored":75487694,"data_compressed":3878781,"data_compressed_allocated":37314560,"data_compressed_original":75128832,"omap_allocated":0,"internal_metadata":55705600},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":73,"seq":313532612713,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":73768,"kb_used_data":33300,"kb_used_omap":0,"kb_used_meta":40448,"kb_avail":94298072,"statfs":{"total":96636764160,"available":96561225728,"internally_reserved":0,"allocated":34099200,"data_stored":67294029,"data_compressed":3455345,"data_compressed_allocated":33206272,"data_compressed_original":66928640,"omap_allocated":0,"internal_metadata":41418752},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29962,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29962,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":33705984,"data_stored":67436544,"data_compressed":3481219,"data_compressed_allocated":33705984,"data_compressed_original":67436544,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":37457920,"data_stored":74924032,"data_compressed":3867854,"data_compressed_allocated":37457920,"data_compressed_original":74924032,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":26869760,"data_stored":53739520,"data_compressed":2774241,"data_compressed_allocated":26869760,"data_compressed_original":53739520,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":32960512,"data_stored":85573632,"data_compressed":4331356,"data_compressed_allocated":32960512,"data_compressed_original":85573632,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-22T03:16:23.785 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-09-22T03:16:23.785 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-22T03:16:23.785 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-09-22T03:16:23.785 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-09-22T03:16:23.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:16:23.914+0000 7f6b6b957640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:16:23.924 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:16:23.922+0000 7f6b6b957640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:16:23.939 INFO:teuthology.orchestra.run.smithi153.stderr:no valid command found; 10 closest matches: 2024-09-22T03:16:23.940 INFO:teuthology.orchestra.run.smithi153.stderr:config get 2024-09-22T03:16:23.940 INFO:teuthology.orchestra.run.smithi153.stderr:config help 2024-09-22T03:16:23.940 INFO:teuthology.orchestra.run.smithi153.stderr:config set ... 2024-09-22T03:16:23.940 INFO:teuthology.orchestra.run.smithi153.stderr:config show 2024-09-22T03:16:23.940 INFO:teuthology.orchestra.run.smithi153.stderr:deep_scrub 2024-09-22T03:16:23.940 INFO:teuthology.orchestra.run.smithi153.stderr:dump_historic_ops 2024-09-22T03:16:23.940 INFO:teuthology.orchestra.run.smithi153.stderr:dump_historic_slow_ops 2024-09-22T03:16:23.940 INFO:teuthology.orchestra.run.smithi153.stderr:dump_metrics [] 2024-09-22T03:16:23.940 INFO:teuthology.orchestra.run.smithi153.stderr:dump_ops_in_flight 2024-09-22T03:16:23.940 INFO:teuthology.orchestra.run.smithi153.stderr:dump_pgstate_history 2024-09-22T03:16:23.940 INFO:teuthology.orchestra.run.smithi153.stderr:Error EINVAL: invalid command 2024-09-22T03:16:23.944 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-22T03:16:23.945 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-22T03:16:23.945 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-09-22T03:16:23.985 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 63 16 12452 12436 1.54201 1.96094 0.0211104 0.0810333 2024-09-22T03:16:24.074 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:16:24.067+0000 7f8becb97640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:16:24.077 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:16:24.076+0000 7f8becb97640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:16:24.091 INFO:teuthology.orchestra.run.smithi153.stderr:no valid command found; 10 closest matches: 2024-09-22T03:16:24.092 INFO:teuthology.orchestra.run.smithi153.stderr:config get 2024-09-22T03:16:24.092 INFO:teuthology.orchestra.run.smithi153.stderr:config help 2024-09-22T03:16:24.092 INFO:teuthology.orchestra.run.smithi153.stderr:config set ... 2024-09-22T03:16:24.092 INFO:teuthology.orchestra.run.smithi153.stderr:config show 2024-09-22T03:16:24.092 INFO:teuthology.orchestra.run.smithi153.stderr:deep_scrub 2024-09-22T03:16:24.092 INFO:teuthology.orchestra.run.smithi153.stderr:dump_historic_ops 2024-09-22T03:16:24.092 INFO:teuthology.orchestra.run.smithi153.stderr:dump_historic_slow_ops 2024-09-22T03:16:24.092 INFO:teuthology.orchestra.run.smithi153.stderr:dump_metrics [] 2024-09-22T03:16:24.092 INFO:teuthology.orchestra.run.smithi153.stderr:dump_ops_in_flight 2024-09-22T03:16:24.092 INFO:teuthology.orchestra.run.smithi153.stderr:dump_pgstate_history 2024-09-22T03:16:24.092 INFO:teuthology.orchestra.run.smithi153.stderr:Error EINVAL: invalid command 2024-09-22T03:16:24.094 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-22T03:16:24.095 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-22T03:16:24.095 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-09-22T03:16:24.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:16:24.255+0000 7f265bc63640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:16:24.268 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:16:24.266+0000 7f265bc63640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:16:24.287 INFO:teuthology.orchestra.run.smithi153.stderr:no valid command found; 10 closest matches: 2024-09-22T03:16:24.287 INFO:teuthology.orchestra.run.smithi153.stderr:config get 2024-09-22T03:16:24.288 INFO:teuthology.orchestra.run.smithi153.stderr:config help 2024-09-22T03:16:24.288 INFO:teuthology.orchestra.run.smithi153.stderr:config set ... 2024-09-22T03:16:24.288 INFO:teuthology.orchestra.run.smithi153.stderr:config show 2024-09-22T03:16:24.288 INFO:teuthology.orchestra.run.smithi153.stderr:deep_scrub 2024-09-22T03:16:24.288 INFO:teuthology.orchestra.run.smithi153.stderr:dump_historic_ops 2024-09-22T03:16:24.288 INFO:teuthology.orchestra.run.smithi153.stderr:dump_historic_slow_ops 2024-09-22T03:16:24.288 INFO:teuthology.orchestra.run.smithi153.stderr:dump_metrics [] 2024-09-22T03:16:24.288 INFO:teuthology.orchestra.run.smithi153.stderr:dump_ops_in_flight 2024-09-22T03:16:24.289 INFO:teuthology.orchestra.run.smithi153.stderr:dump_pgstate_history 2024-09-22T03:16:24.289 INFO:teuthology.orchestra.run.smithi153.stderr:Error EINVAL: invalid command 2024-09-22T03:16:24.290 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-22T03:16:24.290 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-22T03:16:24.291 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-09-22T03:16:24.453 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:16:24.443+0000 7f2d7c357640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:16:24.458 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:16:24.456+0000 7f2d7c357640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:16:24.478 INFO:teuthology.orchestra.run.smithi153.stderr:no valid command found; 10 closest matches: 2024-09-22T03:16:24.478 INFO:teuthology.orchestra.run.smithi153.stderr:config get 2024-09-22T03:16:24.478 INFO:teuthology.orchestra.run.smithi153.stderr:config help 2024-09-22T03:16:24.478 INFO:teuthology.orchestra.run.smithi153.stderr:config set ... 2024-09-22T03:16:24.478 INFO:teuthology.orchestra.run.smithi153.stderr:config show 2024-09-22T03:16:24.478 INFO:teuthology.orchestra.run.smithi153.stderr:deep_scrub 2024-09-22T03:16:24.478 INFO:teuthology.orchestra.run.smithi153.stderr:dump_historic_ops 2024-09-22T03:16:24.479 INFO:teuthology.orchestra.run.smithi153.stderr:dump_historic_slow_ops 2024-09-22T03:16:24.479 INFO:teuthology.orchestra.run.smithi153.stderr:dump_metrics [] 2024-09-22T03:16:24.479 INFO:teuthology.orchestra.run.smithi153.stderr:dump_ops_in_flight 2024-09-22T03:16:24.479 INFO:teuthology.orchestra.run.smithi153.stderr:dump_pgstate_history 2024-09-22T03:16:24.479 INFO:teuthology.orchestra.run.smithi153.stderr:Error EINVAL: invalid command 2024-09-22T03:16:24.482 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-22T03:16:24.482 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-22T03:16:24.986 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 64 15 12723 12708 1.55112 2.125 0.0878874 0.080528 2024-09-22T03:16:24.998 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-22T03:16:25.115 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:16:25.112+0000 7f5d5462c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:16:25.120 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:16:25.118+0000 7f5d5462c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:16:25.986 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 65 16 13018 13002 1.56259 2.29688 0.10088 0.07994 2024-09-22T03:16:26.986 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 66 16 13248 13232 1.56614 1.79688 0.0437297 0.0797802 2024-09-22T03:16:27.360 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-22T03:16:27.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:16:27.489+0000 7fbc762c3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:16:27.499 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:16:27.497+0000 7fbc762c3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:16:27.986 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 67 16 13543 13527 1.57716 2.30469 0.039713 0.0792342 2024-09-22T03:16:28.986 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 68 16 13788 13772 1.58211 1.91406 0.102867 0.0789306 2024-09-22T03:16:29.482 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [1, 0, 3, 2] 2024-09-22T03:16:29.483 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-22T03:16:29.483 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-09-22T03:16:29.483 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-09-22T03:16:29.633 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:16:29.623+0000 7faf41797640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:16:29.637 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:16:29.636+0000 7faf41797640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:16:29.660 INFO:teuthology.orchestra.run.smithi153.stderr:no valid command found; 10 closest matches: 2024-09-22T03:16:29.660 INFO:teuthology.orchestra.run.smithi153.stderr:config get 2024-09-22T03:16:29.661 INFO:teuthology.orchestra.run.smithi153.stderr:config help 2024-09-22T03:16:29.661 INFO:teuthology.orchestra.run.smithi153.stderr:config set ... 2024-09-22T03:16:29.661 INFO:teuthology.orchestra.run.smithi153.stderr:config show 2024-09-22T03:16:29.661 INFO:teuthology.orchestra.run.smithi153.stderr:deep_scrub 2024-09-22T03:16:29.661 INFO:teuthology.orchestra.run.smithi153.stderr:dump_historic_ops 2024-09-22T03:16:29.662 INFO:teuthology.orchestra.run.smithi153.stderr:dump_historic_slow_ops 2024-09-22T03:16:29.662 INFO:teuthology.orchestra.run.smithi153.stderr:dump_metrics [] 2024-09-22T03:16:29.662 INFO:teuthology.orchestra.run.smithi153.stderr:dump_ops_in_flight 2024-09-22T03:16:29.662 INFO:teuthology.orchestra.run.smithi153.stderr:dump_pgstate_history 2024-09-22T03:16:29.662 INFO:teuthology.orchestra.run.smithi153.stderr:Error EINVAL: invalid command 2024-09-22T03:16:29.665 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-22T03:16:29.666 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-22T03:16:29.666 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-09-22T03:16:29.802 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:16:29.793+0000 7f9c8bf17640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:16:29.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:16:29.805+0000 7f9c8bf17640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:16:29.824 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-22T03:16:29.859 INFO:teuthology.orchestra.run.smithi153.stderr:no valid command found; 10 closest matches: 2024-09-22T03:16:29.859 INFO:teuthology.orchestra.run.smithi153.stderr:config get 2024-09-22T03:16:29.859 INFO:teuthology.orchestra.run.smithi153.stderr:config help 2024-09-22T03:16:29.859 INFO:teuthology.orchestra.run.smithi153.stderr:config set ... 2024-09-22T03:16:29.859 INFO:teuthology.orchestra.run.smithi153.stderr:config show 2024-09-22T03:16:29.859 INFO:teuthology.orchestra.run.smithi153.stderr:deep_scrub 2024-09-22T03:16:29.859 INFO:teuthology.orchestra.run.smithi153.stderr:dump_historic_ops 2024-09-22T03:16:29.859 INFO:teuthology.orchestra.run.smithi153.stderr:dump_historic_slow_ops 2024-09-22T03:16:29.859 INFO:teuthology.orchestra.run.smithi153.stderr:dump_metrics [] 2024-09-22T03:16:29.859 INFO:teuthology.orchestra.run.smithi153.stderr:dump_ops_in_flight 2024-09-22T03:16:29.860 INFO:teuthology.orchestra.run.smithi153.stderr:dump_pgstate_history 2024-09-22T03:16:29.860 INFO:teuthology.orchestra.run.smithi153.stderr:Error EINVAL: invalid command 2024-09-22T03:16:29.862 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-22T03:16:29.863 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-22T03:16:29.863 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-09-22T03:16:29.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:16:29.943+0000 7fe1a9597640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:16:29.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:16:29.950+0000 7fe1a9597640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:16:29.986 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 69 16 14052 14036 1.58907 2.0625 0.0961259 0.078618 2024-09-22T03:16:30.035 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:16:30.029+0000 7fc2b4cba640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:16:30.038 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:16:30.037+0000 7fc2b4cba640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:16:30.058 INFO:teuthology.orchestra.run.smithi153.stderr:no valid command found; 10 closest matches: 2024-09-22T03:16:30.058 INFO:teuthology.orchestra.run.smithi153.stderr:config get 2024-09-22T03:16:30.058 INFO:teuthology.orchestra.run.smithi153.stderr:config help 2024-09-22T03:16:30.058 INFO:teuthology.orchestra.run.smithi153.stderr:config set ... 2024-09-22T03:16:30.058 INFO:teuthology.orchestra.run.smithi153.stderr:config show 2024-09-22T03:16:30.058 INFO:teuthology.orchestra.run.smithi153.stderr:deep_scrub 2024-09-22T03:16:30.058 INFO:teuthology.orchestra.run.smithi153.stderr:dump_historic_ops 2024-09-22T03:16:30.058 INFO:teuthology.orchestra.run.smithi153.stderr:dump_historic_slow_ops 2024-09-22T03:16:30.058 INFO:teuthology.orchestra.run.smithi153.stderr:dump_metrics [] 2024-09-22T03:16:30.058 INFO:teuthology.orchestra.run.smithi153.stderr:dump_ops_in_flight 2024-09-22T03:16:30.058 INFO:teuthology.orchestra.run.smithi153.stderr:dump_pgstate_history 2024-09-22T03:16:30.059 INFO:teuthology.orchestra.run.smithi153.stderr:Error EINVAL: invalid command 2024-09-22T03:16:30.060 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-22T03:16:30.061 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-22T03:16:30.061 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-09-22T03:16:30.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:16:30.208+0000 7fc9abe58640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:16:30.219 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:16:30.217+0000 7fc9abe58640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:16:30.272 INFO:teuthology.orchestra.run.smithi153.stderr:no valid command found; 10 closest matches: 2024-09-22T03:16:30.272 INFO:teuthology.orchestra.run.smithi153.stderr:config get 2024-09-22T03:16:30.272 INFO:teuthology.orchestra.run.smithi153.stderr:config help 2024-09-22T03:16:30.272 INFO:teuthology.orchestra.run.smithi153.stderr:config set ... 2024-09-22T03:16:30.272 INFO:teuthology.orchestra.run.smithi153.stderr:config show 2024-09-22T03:16:30.272 INFO:teuthology.orchestra.run.smithi153.stderr:deep_scrub 2024-09-22T03:16:30.272 INFO:teuthology.orchestra.run.smithi153.stderr:dump_historic_ops 2024-09-22T03:16:30.272 INFO:teuthology.orchestra.run.smithi153.stderr:dump_historic_slow_ops 2024-09-22T03:16:30.273 INFO:teuthology.orchestra.run.smithi153.stderr:dump_metrics [] 2024-09-22T03:16:30.273 INFO:teuthology.orchestra.run.smithi153.stderr:dump_ops_in_flight 2024-09-22T03:16:30.273 INFO:teuthology.orchestra.run.smithi153.stderr:dump_pgstate_history 2024-09-22T03:16:30.273 INFO:teuthology.orchestra.run.smithi153.stderr:Error EINVAL: invalid command 2024-09-22T03:16:30.275 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-22T03:16:30.275 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-22T03:16:30.986 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 70 16 14309 14293 1.59505 2.00781 0.0568877 0.0783369 2024-09-22T03:16:31.986 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 71 16 14574 14558 1.60174 2.07031 0.0640482 0.0779919 2024-09-22T03:16:32.140 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-22T03:16:32.271 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:16:32.268+0000 7f97a0d17640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:16:32.275 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:16:32.273+0000 7f97a0d17640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:16:32.986 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 72 16 14842 14826 1.60857 2.09375 0.051074 0.0776688 2024-09-22T03:16:33.986 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 73 16 15116 15100 1.61586 2.14062 0.0696145 0.0773105 2024-09-22T03:16:34.481 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-22T03:16:34.601 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:16:34.597+0000 7f38806d7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:16:34.605 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:16:34.603+0000 7f38806d7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:16:34.986 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 74 16 15400 15384 1.624 2.21875 0.0382103 0.0769387 2024-09-22T03:16:35.276 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [1, 0, 3, 2] 2024-09-22T03:16:35.276 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-09-22T03:16:35.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:16:35.403+0000 7f1bf922c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:16:35.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:16:35.412+0000 7f1bf922c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:16:35.987 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 75 16 15656 15640 1.62901 2 0.0189725 0.0766988 2024-09-22T03:16:36.270 INFO:teuthology.orchestra.run.smithi153.stderr:reweighted osd.0 to 1 (10000) 2024-09-22T03:16:36.291 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-09-22T03:16:36.444 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:16:36.433+0000 7f2e84997640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:16:36.450 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:16:36.448+0000 7f2e84997640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:16:36.765 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-22T03:16:36.905 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:16:36.902+0000 7f5a68b97640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:16:36.910 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:16:36.908+0000 7f5a68b97640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:16:36.987 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 76 16 15923 15907 1.63502 2.08594 0.0237872 0.0764218 2024-09-22T03:16:37.282 INFO:teuthology.orchestra.run.smithi153.stderr:reweighted osd.1 to 1 (10000) 2024-09-22T03:16:37.305 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-09-22T03:16:37.462 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:16:37.452+0000 7f5a1be2c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:16:37.467 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:16:37.465+0000 7f5a1be2c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:16:37.987 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 77 16 16174 16158 1.63925 1.96094 0.0208655 0.0762069 2024-09-22T03:16:38.287 INFO:teuthology.orchestra.run.smithi153.stderr:reweighted osd.2 to 1 (10000) 2024-09-22T03:16:38.303 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-09-22T03:16:38.431 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:16:38.423+0000 7f70b5197640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:16:38.434 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:16:38.432+0000 7f70b5197640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:16:38.912 INFO:teuthology.orchestra.run.smithi153.stderr:reweighted osd.3 to 1 (10000) 2024-09-22T03:16:38.928 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-09-22T03:16:38.928 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-22T03:16:38.987 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 78 16 16415 16399 1.64237 1.88281 0.0401117 0.0760782 2024-09-22T03:16:39.053 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:16:39.048+0000 7fc2ecf17640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:16:39.056 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:16:39.055+0000 7fc2ecf17640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:16:39.137 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-22T03:16:39.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:16:39.295+0000 7fd215ed7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:16:39.302 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:16:39.301+0000 7fd215ed7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:16:39.322 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-22T03:16:39.323 INFO:teuthology.orchestra.run.smithi153.stderr:dumped all 2024-09-22T03:16:39.338 INFO:teuthology.orchestra.run.smithi153.stdout:{"pg_ready":true,"pg_map":{"version":361,"stamp":"2024-09-22T03:16:38.911932+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":127418931,"num_objects":1940,"num_object_clones":0,"num_object_copies":5820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":15621,"num_write_kb":125752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":322,"num_bytes_recovered":21020672,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":15569,"ondisk_log_size":15569,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":348664,"kb_used_data":178168,"kb_used_omap":0,"kb_used_meta":170432,"kb_avail":377138696,"statfs":{"total":386547056640,"available":386190024704,"internally_reserved":0,"allocated":182444032,"data_stored":380247646,"data_compressed":19469545,"data_compressed_allocated":178806784,"data_compressed_original":378773504,"omap_allocated":0,"internal_metadata":174522368},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":4669440,"num_objects":72,"num_object_clones":0,"num_object_copies":216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":570,"num_write_kb":4560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.696084"},"pg_stats":[{"pgid":"7.f","version":"106'983","reported_seq":1065,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-22T03:16:34.083787+0000","last_change":"2024-09-22T03:15:33.972925+0000","last_active":"2024-09-22T03:16:34.083787+0000","last_peered":"2024-09-22T03:16:34.083787+0000","last_clean":"2024-09-22T03:16:34.083787+0000","last_became_active":"2024-09-22T03:15:33.967199+0000","last_became_peered":"2024-09-22T03:15:33.967199+0000","last_unstale":"2024-09-22T03:16:34.083787+0000","last_undegraded":"2024-09-22T03:16:34.083787+0000","last_fullsized":"2024-09-22T03:16:34.083787+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_clean_scrub_stamp":"2024-09-22T03:15:18.897706+0000","objects_scrubbed":0,"log_size":983,"log_dups_size":0,"ondisk_log_size":983,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8060928,"num_objects":123,"num_object_clones":0,"num_object_copies":369,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":984,"num_write_kb":7872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":1245184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.e","version":"106'959","reported_seq":1033,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-22T03:16:33.869239+0000","last_change":"2024-09-22T03:16:18.028856+0000","last_active":"2024-09-22T03:16:33.869239+0000","last_peered":"2024-09-22T03:16:33.869239+0000","last_clean":"2024-09-22T03:16:33.869239+0000","last_became_active":"2024-09-22T03:16:18.021495+0000","last_became_peered":"2024-09-22T03:16:18.021495+0000","last_unstale":"2024-09-22T03:16:33.869239+0000","last_undegraded":"2024-09-22T03:16:33.869239+0000","last_fullsized":"2024-09-22T03:16:33.869239+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_clean_scrub_stamp":"2024-09-22T03:15:18.897706+0000","objects_scrubbed":0,"log_size":959,"log_dups_size":0,"ondisk_log_size":959,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":960,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.d","version":"107'1071","reported_seq":1204,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-22T03:16:36.547152+0000","last_change":"2024-09-22T03:16:18.042408+0000","last_active":"2024-09-22T03:16:36.547152+0000","last_peered":"2024-09-22T03:16:36.547152+0000","last_clean":"2024-09-22T03:16:36.547152+0000","last_became_active":"2024-09-22T03:16:18.029899+0000","last_became_peered":"2024-09-22T03:16:18.029899+0000","last_unstale":"2024-09-22T03:16:36.547152+0000","last_undegraded":"2024-09-22T03:16:36.547152+0000","last_fullsized":"2024-09-22T03:16:36.547152+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_clean_scrub_stamp":"2024-09-22T03:15:18.897706+0000","objects_scrubbed":0,"log_size":1071,"log_dups_size":0,"ondisk_log_size":1071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8781824,"num_objects":134,"num_object_clones":0,"num_object_copies":402,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1072,"num_write_kb":8576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":1835008,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.c","version":"107'1081","reported_seq":1190,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-22T03:16:36.677858+0000","last_change":"2024-09-22T03:16:18.031863+0000","last_active":"2024-09-22T03:16:36.677858+0000","last_peered":"2024-09-22T03:16:36.677858+0000","last_clean":"2024-09-22T03:16:36.677858+0000","last_became_active":"2024-09-22T03:16:18.022365+0000","last_became_peered":"2024-09-22T03:16:18.022365+0000","last_unstale":"2024-09-22T03:16:36.677858+0000","last_undegraded":"2024-09-22T03:16:36.677858+0000","last_fullsized":"2024-09-22T03:16:36.677858+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_clean_scrub_stamp":"2024-09-22T03:15:18.897706+0000","objects_scrubbed":0,"log_size":1081,"log_dups_size":0,"ondisk_log_size":1081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8863744,"num_objects":136,"num_object_clones":0,"num_object_copies":408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1082,"num_write_kb":8656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":1048576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.b","version":"106'895","reported_seq":956,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-22T03:16:33.617666+0000","last_change":"2024-09-22T03:15:34.925809+0000","last_active":"2024-09-22T03:16:33.617666+0000","last_peered":"2024-09-22T03:16:33.617666+0000","last_clean":"2024-09-22T03:16:33.617666+0000","last_became_active":"2024-09-22T03:15:34.920855+0000","last_became_peered":"2024-09-22T03:15:34.920855+0000","last_unstale":"2024-09-22T03:16:33.617666+0000","last_undegraded":"2024-09-22T03:16:33.617666+0000","last_fullsized":"2024-09-22T03:16:33.617666+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_clean_scrub_stamp":"2024-09-22T03:15:18.897706+0000","objects_scrubbed":0,"log_size":895,"log_dups_size":0,"ondisk_log_size":895,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7340032,"num_objects":112,"num_object_clones":0,"num_object_copies":336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":896,"num_write_kb":7168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":2031616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"106'887","reported_seq":952,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-22T03:16:33.977290+0000","last_change":"2024-09-22T03:16:18.015447+0000","last_active":"2024-09-22T03:16:33.977290+0000","last_peered":"2024-09-22T03:16:33.977290+0000","last_clean":"2024-09-22T03:16:33.977290+0000","last_became_active":"2024-09-22T03:16:18.012158+0000","last_became_peered":"2024-09-22T03:16:18.012158+0000","last_unstale":"2024-09-22T03:16:33.977290+0000","last_undegraded":"2024-09-22T03:16:33.977290+0000","last_fullsized":"2024-09-22T03:16:33.977290+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_clean_scrub_stamp":"2024-09-22T03:15:18.897706+0000","objects_scrubbed":0,"log_size":887,"log_dups_size":0,"ondisk_log_size":887,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7274496,"num_objects":111,"num_object_clones":0,"num_object_copies":333,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":888,"num_write_kb":7104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":1507328,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.9","version":"106'880","reported_seq":992,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-22T03:16:36.280176+0000","last_change":"2024-09-22T03:16:18.095024+0000","last_active":"2024-09-22T03:16:36.280176+0000","last_peered":"2024-09-22T03:16:36.280176+0000","last_clean":"2024-09-22T03:16:36.280176+0000","last_became_active":"2024-09-22T03:16:18.088681+0000","last_became_peered":"2024-09-22T03:16:18.088681+0000","last_unstale":"2024-09-22T03:16:36.280176+0000","last_undegraded":"2024-09-22T03:16:36.280176+0000","last_fullsized":"2024-09-22T03:16:36.280176+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_clean_scrub_stamp":"2024-09-22T03:15:18.897706+0000","objects_scrubbed":0,"log_size":880,"log_dups_size":0,"ondisk_log_size":880,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7208960,"num_objects":110,"num_object_clones":0,"num_object_copies":330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":880,"num_write_kb":7040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":1179648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.8","version":"106'927","reported_seq":1047,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-22T03:16:33.273616+0000","last_change":"2024-09-22T03:15:33.946927+0000","last_active":"2024-09-22T03:16:33.273616+0000","last_peered":"2024-09-22T03:16:33.273616+0000","last_clean":"2024-09-22T03:16:33.273616+0000","last_became_active":"2024-09-22T03:15:33.940267+0000","last_became_peered":"2024-09-22T03:15:33.940267+0000","last_unstale":"2024-09-22T03:16:33.273616+0000","last_undegraded":"2024-09-22T03:16:33.273616+0000","last_fullsized":"2024-09-22T03:16:33.273616+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_clean_scrub_stamp":"2024-09-22T03:15:18.897706+0000","objects_scrubbed":0,"log_size":927,"log_dups_size":0,"ondisk_log_size":927,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7602176,"num_objects":116,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":928,"num_write_kb":7424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":1507328,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.6","version":"106'1023","reported_seq":1116,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-22T03:16:34.025857+0000","last_change":"2024-09-22T03:15:32.999617+0000","last_active":"2024-09-22T03:16:34.025857+0000","last_peered":"2024-09-22T03:16:34.025857+0000","last_clean":"2024-09-22T03:16:34.025857+0000","last_became_active":"2024-09-22T03:15:32.993508+0000","last_became_peered":"2024-09-22T03:15:32.993508+0000","last_unstale":"2024-09-22T03:16:34.025857+0000","last_undegraded":"2024-09-22T03:16:34.025857+0000","last_fullsized":"2024-09-22T03:16:34.025857+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_clean_scrub_stamp":"2024-09-22T03:15:18.897706+0000","objects_scrubbed":0,"log_size":1023,"log_dups_size":0,"ondisk_log_size":1023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":128,"num_object_clones":0,"num_object_copies":384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1024,"num_write_kb":8192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":491520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":156,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-22T03:16:22.966948+0000","last_change":"2024-09-22T03:16:18.969989+0000","last_active":"2024-09-22T03:16:22.966948+0000","last_peered":"2024-09-22T03:16:22.966948+0000","last_clean":"2024-09-22T03:16:22.966948+0000","last_became_active":"2024-09-22T03:16:18.003684+0000","last_became_peered":"2024-09-22T03:16:18.003684+0000","last_unstale":"2024-09-22T03:16:22.966948+0000","last_undegraded":"2024-09-22T03:16:22.966948+0000","last_fullsized":"2024-09-22T03:16:22.966948+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.7","version":"106'1119","reported_seq":1249,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-22T03:16:33.591419+0000","last_change":"2024-09-22T03:15:33.949109+0000","last_active":"2024-09-22T03:16:33.591419+0000","last_peered":"2024-09-22T03:16:33.591419+0000","last_clean":"2024-09-22T03:16:33.591419+0000","last_became_active":"2024-09-22T03:15:33.943481+0000","last_became_peered":"2024-09-22T03:15:33.943481+0000","last_unstale":"2024-09-22T03:16:33.591419+0000","last_undegraded":"2024-09-22T03:16:33.591419+0000","last_fullsized":"2024-09-22T03:16:33.591419+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_clean_scrub_stamp":"2024-09-22T03:15:18.897706+0000","objects_scrubbed":0,"log_size":1119,"log_dups_size":0,"ondisk_log_size":1119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9175040,"num_objects":140,"num_object_clones":0,"num_object_copies":420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1120,"num_write_kb":8960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":1835008,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":242,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-22T03:16:36.308218+0000","last_change":"2024-09-22T03:16:18.925677+0000","last_active":"2024-09-22T03:16:36.308218+0000","last_peered":"2024-09-22T03:16:36.308218+0000","last_clean":"2024-09-22T03:16:36.308218+0000","last_became_active":"2024-09-22T03:16:17.986660+0000","last_became_peered":"2024-09-22T03:16:17.986660+0000","last_unstale":"2024-09-22T03:16:36.308218+0000","last_undegraded":"2024-09-22T03:16:36.308218+0000","last_fullsized":"2024-09-22T03:16:36.308218+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.4","version":"106'1040","reported_seq":1158,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-22T03:16:36.278873+0000","last_change":"2024-09-22T03:16:18.072690+0000","last_active":"2024-09-22T03:16:36.278873+0000","last_peered":"2024-09-22T03:16:36.278873+0000","last_clean":"2024-09-22T03:16:36.278873+0000","last_became_active":"2024-09-22T03:16:18.063063+0000","last_became_peered":"2024-09-22T03:16:18.063063+0000","last_unstale":"2024-09-22T03:16:36.278873+0000","last_undegraded":"2024-09-22T03:16:36.278873+0000","last_fullsized":"2024-09-22T03:16:36.278873+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_clean_scrub_stamp":"2024-09-22T03:15:18.897706+0000","objects_scrubbed":0,"log_size":1040,"log_dups_size":0,"ondisk_log_size":1040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8519680,"num_objects":130,"num_object_clones":0,"num_object_copies":390,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1040,"num_write_kb":8320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1376256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":239,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-22T03:16:36.281284+0000","last_change":"2024-09-22T03:16:18.926944+0000","last_active":"2024-09-22T03:16:36.281284+0000","last_peered":"2024-09-22T03:16:36.281284+0000","last_clean":"2024-09-22T03:16:36.281284+0000","last_became_active":"2024-09-22T03:16:18.083437+0000","last_became_peered":"2024-09-22T03:16:18.083437+0000","last_unstale":"2024-09-22T03:16:36.281284+0000","last_undegraded":"2024-09-22T03:16:36.281284+0000","last_fullsized":"2024-09-22T03:16:36.281284+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.5","version":"106'903","reported_seq":1039,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-22T03:16:33.900283+0000","last_change":"2024-09-22T03:16:18.004962+0000","last_active":"2024-09-22T03:16:33.900283+0000","last_peered":"2024-09-22T03:16:33.900283+0000","last_clean":"2024-09-22T03:16:33.900283+0000","last_became_active":"2024-09-22T03:16:18.000028+0000","last_became_peered":"2024-09-22T03:16:18.000028+0000","last_unstale":"2024-09-22T03:16:33.900283+0000","last_undegraded":"2024-09-22T03:16:33.900283+0000","last_fullsized":"2024-09-22T03:16:33.900283+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_clean_scrub_stamp":"2024-09-22T03:15:18.897706+0000","objects_scrubbed":0,"log_size":903,"log_dups_size":0,"ondisk_log_size":903,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7405568,"num_objects":113,"num_object_clones":0,"num_object_copies":339,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":904,"num_write_kb":7232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":1966080,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":215,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-22T03:16:22.964343+0000","last_change":"2024-09-22T03:15:30.924363+0000","last_active":"2024-09-22T03:16:22.964343+0000","last_peered":"2024-09-22T03:16:22.964343+0000","last_clean":"2024-09-22T03:16:22.964343+0000","last_became_active":"2024-09-22T03:15:29.958549+0000","last_became_peered":"2024-09-22T03:15:29.958549+0000","last_unstale":"2024-09-22T03:16:22.964343+0000","last_undegraded":"2024-09-22T03:16:22.964343+0000","last_fullsized":"2024-09-22T03:16:22.964343+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"12'97","reported_seq":364,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-22T03:16:36.316682+0000","last_change":"2024-09-22T03:16:17.998607+0000","last_active":"2024-09-22T03:16:36.316682+0000","last_peered":"2024-09-22T03:16:36.316682+0000","last_clean":"2024-09-22T03:16:36.316682+0000","last_became_active":"2024-09-22T03:16:17.995046+0000","last_became_peered":"2024-09-22T03:16:17.995046+0000","last_unstale":"2024-09-22T03:16:36.316682+0000","last_undegraded":"2024-09-22T03:16:36.316682+0000","last_fullsized":"2024-09-22T03:16:36.316682+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.2","version":"106'919","reported_seq":1007,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-22T03:16:33.767173+0000","last_change":"2024-09-22T03:15:33.953595+0000","last_active":"2024-09-22T03:16:33.767173+0000","last_peered":"2024-09-22T03:16:33.767173+0000","last_clean":"2024-09-22T03:16:33.767173+0000","last_became_active":"2024-09-22T03:15:33.947435+0000","last_became_peered":"2024-09-22T03:15:33.947435+0000","last_unstale":"2024-09-22T03:16:33.767173+0000","last_undegraded":"2024-09-22T03:16:33.767173+0000","last_fullsized":"2024-09-22T03:16:33.767173+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_clean_scrub_stamp":"2024-09-22T03:15:18.897706+0000","objects_scrubbed":0,"log_size":919,"log_dups_size":0,"ondisk_log_size":919,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7536640,"num_objects":115,"num_object_clones":0,"num_object_copies":345,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":920,"num_write_kb":7360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1376256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":244,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-22T03:16:22.968835+0000","last_change":"2024-09-22T03:16:18.925553+0000","last_active":"2024-09-22T03:16:22.968835+0000","last_peered":"2024-09-22T03:16:22.968835+0000","last_clean":"2024-09-22T03:16:22.968835+0000","last_became_active":"2024-09-22T03:16:18.021400+0000","last_became_peered":"2024-09-22T03:16:18.021400+0000","last_unstale":"2024-09-22T03:16:22.968835+0000","last_undegraded":"2024-09-22T03:16:22.968835+0000","last_fullsized":"2024-09-22T03:16:22.968835+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.3","version":"106'919","reported_seq":1004,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-22T03:16:32.416964+0000","last_change":"2024-09-22T03:16:18.004775+0000","last_active":"2024-09-22T03:16:32.416964+0000","last_peered":"2024-09-22T03:16:32.416964+0000","last_clean":"2024-09-22T03:16:32.416964+0000","last_became_active":"2024-09-22T03:16:17.999612+0000","last_became_peered":"2024-09-22T03:16:17.999612+0000","last_unstale":"2024-09-22T03:16:32.416964+0000","last_undegraded":"2024-09-22T03:16:32.416964+0000","last_fullsized":"2024-09-22T03:16:32.416964+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_clean_scrub_stamp":"2024-09-22T03:15:18.897706+0000","objects_scrubbed":0,"log_size":919,"log_dups_size":0,"ondisk_log_size":919,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7536640,"num_objects":115,"num_object_clones":0,"num_object_copies":345,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":920,"num_write_kb":7360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1376256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":218,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-22T03:16:36.280703+0000","last_change":"2024-09-22T03:16:18.927373+0000","last_active":"2024-09-22T03:16:36.280703+0000","last_peered":"2024-09-22T03:16:36.280703+0000","last_clean":"2024-09-22T03:16:36.280703+0000","last_became_active":"2024-09-22T03:16:18.086190+0000","last_became_peered":"2024-09-22T03:16:18.086190+0000","last_unstale":"2024-09-22T03:16:36.280703+0000","last_undegraded":"2024-09-22T03:16:36.280703+0000","last_fullsized":"2024-09-22T03:16:36.280703+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.0","version":"106'968","reported_seq":1089,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-22T03:16:36.277367+0000","last_change":"2024-09-22T03:16:18.066854+0000","last_active":"2024-09-22T03:16:36.277367+0000","last_peered":"2024-09-22T03:16:36.277367+0000","last_clean":"2024-09-22T03:16:36.277367+0000","last_became_active":"2024-09-22T03:16:18.055858+0000","last_became_peered":"2024-09-22T03:16:18.055858+0000","last_unstale":"2024-09-22T03:16:36.277367+0000","last_undegraded":"2024-09-22T03:16:36.277367+0000","last_fullsized":"2024-09-22T03:16:36.277367+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_clean_scrub_stamp":"2024-09-22T03:15:18.897706+0000","objects_scrubbed":0,"log_size":968,"log_dups_size":0,"ondisk_log_size":968,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7929856,"num_objects":121,"num_object_clones":0,"num_object_copies":363,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":968,"num_write_kb":7744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1327104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":258,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-22T03:16:22.969675+0000","last_change":"2024-09-22T03:16:18.928041+0000","last_active":"2024-09-22T03:16:22.969675+0000","last_peered":"2024-09-22T03:16:22.969675+0000","last_clean":"2024-09-22T03:16:22.969675+0000","last_became_active":"2024-09-22T03:16:18.025488+0000","last_became_peered":"2024-09-22T03:16:18.025488+0000","last_unstale":"2024-09-22T03:16:22.969675+0000","last_undegraded":"2024-09-22T03:16:22.969675+0000","last_fullsized":"2024-09-22T03:16:22.969675+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1","version":"107'895","reported_seq":1000,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-22T03:16:36.681448+0000","last_change":"2024-09-22T03:15:33.950769+0000","last_active":"2024-09-22T03:16:36.681448+0000","last_peered":"2024-09-22T03:16:36.681448+0000","last_clean":"2024-09-22T03:16:36.681448+0000","last_became_active":"2024-09-22T03:15:33.945111+0000","last_became_peered":"2024-09-22T03:15:33.945111+0000","last_unstale":"2024-09-22T03:16:36.681448+0000","last_undegraded":"2024-09-22T03:16:36.681448+0000","last_fullsized":"2024-09-22T03:16:36.681448+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_clean_scrub_stamp":"2024-09-22T03:15:18.897706+0000","objects_scrubbed":0,"log_size":895,"log_dups_size":0,"ondisk_log_size":895,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7340032,"num_objects":112,"num_object_clones":0,"num_object_copies":336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":896,"num_write_kb":7168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":917504,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":251,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-22T03:16:22.970296+0000","last_change":"2024-09-22T03:15:30.958189+0000","last_active":"2024-09-22T03:16:22.970296+0000","last_peered":"2024-09-22T03:16:22.970296+0000","last_clean":"2024-09-22T03:16:22.970296+0000","last_became_active":"2024-09-22T03:15:29.980027+0000","last_became_peered":"2024-09-22T03:15:29.980027+0000","last_unstale":"2024-09-22T03:16:22.970296+0000","last_undegraded":"2024-09-22T03:16:22.970296+0000","last_fullsized":"2024-09-22T03:16:22.970296+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":126828544,"num_objects":1936,"num_object_clones":0,"num_object_copies":5808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15482,"num_write_kb":123856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":322,"num_bytes_recovered":21020672,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":179777536,"data_stored":379240448,"data_compressed":19492600,"data_compressed_allocated":179777536,"data_compressed_original":379240448,"omap_allocated":0,"internal_metadata":0},"log_size":15469,"ondisk_log_size":15469,"up":48,"acting":48,"num_store_stats":4},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1180736,"data_compressed":59924,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":87,"seq":373662154817,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":81144,"kb_used_data":47848,"kb_used_omap":0,"kb_used_meta":33280,"kb_avail":94290696,"statfs":{"total":96636764160,"available":96553672704,"internally_reserved":0,"allocated":48996352,"data_stored":116214482,"data_compressed":5895187,"data_compressed_allocated":48103424,"data_compressed_original":115859456,"omap_allocated":0,"internal_metadata":34078720},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":100,"seq":429496729619,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":68404,"kb_used_data":35812,"kb_used_omap":0,"kb_used_meta":32576,"kb_avail":94303436,"statfs":{"total":96636764160,"available":96566718464,"internally_reserved":0,"allocated":36671488,"data_stored":72379634,"data_compressed":3717883,"data_compressed_allocated":35758080,"data_compressed_original":72007680,"omap_allocated":0,"internal_metadata":33357824},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":53,"seq":227633266927,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":100500,"kb_used_data":50244,"kb_used_omap":0,"kb_used_meta":50240,"kb_avail":94271340,"statfs":{"total":96636764160,"available":96533852160,"internally_reserved":0,"allocated":51449856,"data_stored":101944562,"data_compressed":5244225,"data_compressed_allocated":50536448,"data_compressed_original":101572608,"omap_allocated":0,"internal_metadata":51445760},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":73,"seq":313532612728,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":98616,"kb_used_data":44264,"kb_used_omap":0,"kb_used_meta":54336,"kb_avail":94273224,"statfs":{"total":96636764160,"available":96535781376,"internally_reserved":0,"allocated":45326336,"data_stored":89708968,"data_compressed":4612250,"data_compressed_allocated":44408832,"data_compressed_original":89333760,"omap_allocated":0,"internal_metadata":55640064},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29962,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29962,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":44802048,"data_stored":89628672,"data_compressed":4627126,"data_compressed_allocated":44802048,"data_compressed_original":89628672,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":50671616,"data_stored":101351424,"data_compressed":5232452,"data_compressed_allocated":50671616,"data_compressed_original":101351424,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36057088,"data_stored":72114176,"data_compressed":3723030,"data_compressed_allocated":36057088,"data_compressed_original":72114176,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":48246784,"data_stored":116146176,"data_compressed":5909992,"data_compressed_allocated":48246784,"data_compressed_original":116146176,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-22T03:16:39.340 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-22T03:16:39.458 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:16:39.451+0000 7fb858917640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:16:39.463 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:16:39.462+0000 7fb858917640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:16:39.789 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-22T03:16:39.790 INFO:teuthology.orchestra.run.smithi153.stderr:dumped all 2024-09-22T03:16:39.808 INFO:teuthology.orchestra.run.smithi153.stdout:{"pg_ready":true,"pg_map":{"version":361,"stamp":"2024-09-22T03:16:38.911932+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":127418931,"num_objects":1940,"num_object_clones":0,"num_object_copies":5820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":15621,"num_write_kb":125752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":322,"num_bytes_recovered":21020672,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":15569,"ondisk_log_size":15569,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":348664,"kb_used_data":178168,"kb_used_omap":0,"kb_used_meta":170432,"kb_avail":377138696,"statfs":{"total":386547056640,"available":386190024704,"internally_reserved":0,"allocated":182444032,"data_stored":380247646,"data_compressed":19469545,"data_compressed_allocated":178806784,"data_compressed_original":378773504,"omap_allocated":0,"internal_metadata":174522368},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":4669440,"num_objects":72,"num_object_clones":0,"num_object_copies":216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":570,"num_write_kb":4560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.696084"},"pg_stats":[{"pgid":"7.f","version":"106'983","reported_seq":1065,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-22T03:16:34.083787+0000","last_change":"2024-09-22T03:15:33.972925+0000","last_active":"2024-09-22T03:16:34.083787+0000","last_peered":"2024-09-22T03:16:34.083787+0000","last_clean":"2024-09-22T03:16:34.083787+0000","last_became_active":"2024-09-22T03:15:33.967199+0000","last_became_peered":"2024-09-22T03:15:33.967199+0000","last_unstale":"2024-09-22T03:16:34.083787+0000","last_undegraded":"2024-09-22T03:16:34.083787+0000","last_fullsized":"2024-09-22T03:16:34.083787+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_clean_scrub_stamp":"2024-09-22T03:15:18.897706+0000","objects_scrubbed":0,"log_size":983,"log_dups_size":0,"ondisk_log_size":983,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8060928,"num_objects":123,"num_object_clones":0,"num_object_copies":369,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":984,"num_write_kb":7872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":1245184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.e","version":"106'959","reported_seq":1033,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-22T03:16:33.869239+0000","last_change":"2024-09-22T03:16:18.028856+0000","last_active":"2024-09-22T03:16:33.869239+0000","last_peered":"2024-09-22T03:16:33.869239+0000","last_clean":"2024-09-22T03:16:33.869239+0000","last_became_active":"2024-09-22T03:16:18.021495+0000","last_became_peered":"2024-09-22T03:16:18.021495+0000","last_unstale":"2024-09-22T03:16:33.869239+0000","last_undegraded":"2024-09-22T03:16:33.869239+0000","last_fullsized":"2024-09-22T03:16:33.869239+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_clean_scrub_stamp":"2024-09-22T03:15:18.897706+0000","objects_scrubbed":0,"log_size":959,"log_dups_size":0,"ondisk_log_size":959,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":960,"num_write_kb":7680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.d","version":"107'1071","reported_seq":1204,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-22T03:16:36.547152+0000","last_change":"2024-09-22T03:16:18.042408+0000","last_active":"2024-09-22T03:16:36.547152+0000","last_peered":"2024-09-22T03:16:36.547152+0000","last_clean":"2024-09-22T03:16:36.547152+0000","last_became_active":"2024-09-22T03:16:18.029899+0000","last_became_peered":"2024-09-22T03:16:18.029899+0000","last_unstale":"2024-09-22T03:16:36.547152+0000","last_undegraded":"2024-09-22T03:16:36.547152+0000","last_fullsized":"2024-09-22T03:16:36.547152+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_clean_scrub_stamp":"2024-09-22T03:15:18.897706+0000","objects_scrubbed":0,"log_size":1071,"log_dups_size":0,"ondisk_log_size":1071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8781824,"num_objects":134,"num_object_clones":0,"num_object_copies":402,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1072,"num_write_kb":8576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":1835008,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.c","version":"107'1081","reported_seq":1190,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-22T03:16:36.677858+0000","last_change":"2024-09-22T03:16:18.031863+0000","last_active":"2024-09-22T03:16:36.677858+0000","last_peered":"2024-09-22T03:16:36.677858+0000","last_clean":"2024-09-22T03:16:36.677858+0000","last_became_active":"2024-09-22T03:16:18.022365+0000","last_became_peered":"2024-09-22T03:16:18.022365+0000","last_unstale":"2024-09-22T03:16:36.677858+0000","last_undegraded":"2024-09-22T03:16:36.677858+0000","last_fullsized":"2024-09-22T03:16:36.677858+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_clean_scrub_stamp":"2024-09-22T03:15:18.897706+0000","objects_scrubbed":0,"log_size":1081,"log_dups_size":0,"ondisk_log_size":1081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8863744,"num_objects":136,"num_object_clones":0,"num_object_copies":408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1082,"num_write_kb":8656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":1048576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.b","version":"106'895","reported_seq":956,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-22T03:16:33.617666+0000","last_change":"2024-09-22T03:15:34.925809+0000","last_active":"2024-09-22T03:16:33.617666+0000","last_peered":"2024-09-22T03:16:33.617666+0000","last_clean":"2024-09-22T03:16:33.617666+0000","last_became_active":"2024-09-22T03:15:34.920855+0000","last_became_peered":"2024-09-22T03:15:34.920855+0000","last_unstale":"2024-09-22T03:16:33.617666+0000","last_undegraded":"2024-09-22T03:16:33.617666+0000","last_fullsized":"2024-09-22T03:16:33.617666+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_clean_scrub_stamp":"2024-09-22T03:15:18.897706+0000","objects_scrubbed":0,"log_size":895,"log_dups_size":0,"ondisk_log_size":895,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7340032,"num_objects":112,"num_object_clones":0,"num_object_copies":336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":896,"num_write_kb":7168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":2031616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"106'887","reported_seq":952,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-22T03:16:33.977290+0000","last_change":"2024-09-22T03:16:18.015447+0000","last_active":"2024-09-22T03:16:33.977290+0000","last_peered":"2024-09-22T03:16:33.977290+0000","last_clean":"2024-09-22T03:16:33.977290+0000","last_became_active":"2024-09-22T03:16:18.012158+0000","last_became_peered":"2024-09-22T03:16:18.012158+0000","last_unstale":"2024-09-22T03:16:33.977290+0000","last_undegraded":"2024-09-22T03:16:33.977290+0000","last_fullsized":"2024-09-22T03:16:33.977290+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_clean_scrub_stamp":"2024-09-22T03:15:18.897706+0000","objects_scrubbed":0,"log_size":887,"log_dups_size":0,"ondisk_log_size":887,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7274496,"num_objects":111,"num_object_clones":0,"num_object_copies":333,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":888,"num_write_kb":7104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":1507328,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.9","version":"106'880","reported_seq":992,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-22T03:16:36.280176+0000","last_change":"2024-09-22T03:16:18.095024+0000","last_active":"2024-09-22T03:16:36.280176+0000","last_peered":"2024-09-22T03:16:36.280176+0000","last_clean":"2024-09-22T03:16:36.280176+0000","last_became_active":"2024-09-22T03:16:18.088681+0000","last_became_peered":"2024-09-22T03:16:18.088681+0000","last_unstale":"2024-09-22T03:16:36.280176+0000","last_undegraded":"2024-09-22T03:16:36.280176+0000","last_fullsized":"2024-09-22T03:16:36.280176+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_clean_scrub_stamp":"2024-09-22T03:15:18.897706+0000","objects_scrubbed":0,"log_size":880,"log_dups_size":0,"ondisk_log_size":880,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7208960,"num_objects":110,"num_object_clones":0,"num_object_copies":330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":880,"num_write_kb":7040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":1179648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.8","version":"106'927","reported_seq":1047,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-22T03:16:33.273616+0000","last_change":"2024-09-22T03:15:33.946927+0000","last_active":"2024-09-22T03:16:33.273616+0000","last_peered":"2024-09-22T03:16:33.273616+0000","last_clean":"2024-09-22T03:16:33.273616+0000","last_became_active":"2024-09-22T03:15:33.940267+0000","last_became_peered":"2024-09-22T03:15:33.940267+0000","last_unstale":"2024-09-22T03:16:33.273616+0000","last_undegraded":"2024-09-22T03:16:33.273616+0000","last_fullsized":"2024-09-22T03:16:33.273616+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_clean_scrub_stamp":"2024-09-22T03:15:18.897706+0000","objects_scrubbed":0,"log_size":927,"log_dups_size":0,"ondisk_log_size":927,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7602176,"num_objects":116,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":928,"num_write_kb":7424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":1507328,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.6","version":"106'1023","reported_seq":1116,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-22T03:16:34.025857+0000","last_change":"2024-09-22T03:15:32.999617+0000","last_active":"2024-09-22T03:16:34.025857+0000","last_peered":"2024-09-22T03:16:34.025857+0000","last_clean":"2024-09-22T03:16:34.025857+0000","last_became_active":"2024-09-22T03:15:32.993508+0000","last_became_peered":"2024-09-22T03:15:32.993508+0000","last_unstale":"2024-09-22T03:16:34.025857+0000","last_undegraded":"2024-09-22T03:16:34.025857+0000","last_fullsized":"2024-09-22T03:16:34.025857+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_clean_scrub_stamp":"2024-09-22T03:15:18.897706+0000","objects_scrubbed":0,"log_size":1023,"log_dups_size":0,"ondisk_log_size":1023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":128,"num_object_clones":0,"num_object_copies":384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1024,"num_write_kb":8192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":491520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":156,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-22T03:16:22.966948+0000","last_change":"2024-09-22T03:16:18.969989+0000","last_active":"2024-09-22T03:16:22.966948+0000","last_peered":"2024-09-22T03:16:22.966948+0000","last_clean":"2024-09-22T03:16:22.966948+0000","last_became_active":"2024-09-22T03:16:18.003684+0000","last_became_peered":"2024-09-22T03:16:18.003684+0000","last_unstale":"2024-09-22T03:16:22.966948+0000","last_undegraded":"2024-09-22T03:16:22.966948+0000","last_fullsized":"2024-09-22T03:16:22.966948+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.7","version":"106'1119","reported_seq":1249,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-22T03:16:33.591419+0000","last_change":"2024-09-22T03:15:33.949109+0000","last_active":"2024-09-22T03:16:33.591419+0000","last_peered":"2024-09-22T03:16:33.591419+0000","last_clean":"2024-09-22T03:16:33.591419+0000","last_became_active":"2024-09-22T03:15:33.943481+0000","last_became_peered":"2024-09-22T03:15:33.943481+0000","last_unstale":"2024-09-22T03:16:33.591419+0000","last_undegraded":"2024-09-22T03:16:33.591419+0000","last_fullsized":"2024-09-22T03:16:33.591419+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_clean_scrub_stamp":"2024-09-22T03:15:18.897706+0000","objects_scrubbed":0,"log_size":1119,"log_dups_size":0,"ondisk_log_size":1119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9175040,"num_objects":140,"num_object_clones":0,"num_object_copies":420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1120,"num_write_kb":8960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":1835008,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":242,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-22T03:16:36.308218+0000","last_change":"2024-09-22T03:16:18.925677+0000","last_active":"2024-09-22T03:16:36.308218+0000","last_peered":"2024-09-22T03:16:36.308218+0000","last_clean":"2024-09-22T03:16:36.308218+0000","last_became_active":"2024-09-22T03:16:17.986660+0000","last_became_peered":"2024-09-22T03:16:17.986660+0000","last_unstale":"2024-09-22T03:16:36.308218+0000","last_undegraded":"2024-09-22T03:16:36.308218+0000","last_fullsized":"2024-09-22T03:16:36.308218+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.4","version":"106'1040","reported_seq":1158,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-22T03:16:36.278873+0000","last_change":"2024-09-22T03:16:18.072690+0000","last_active":"2024-09-22T03:16:36.278873+0000","last_peered":"2024-09-22T03:16:36.278873+0000","last_clean":"2024-09-22T03:16:36.278873+0000","last_became_active":"2024-09-22T03:16:18.063063+0000","last_became_peered":"2024-09-22T03:16:18.063063+0000","last_unstale":"2024-09-22T03:16:36.278873+0000","last_undegraded":"2024-09-22T03:16:36.278873+0000","last_fullsized":"2024-09-22T03:16:36.278873+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_clean_scrub_stamp":"2024-09-22T03:15:18.897706+0000","objects_scrubbed":0,"log_size":1040,"log_dups_size":0,"ondisk_log_size":1040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8519680,"num_objects":130,"num_object_clones":0,"num_object_copies":390,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1040,"num_write_kb":8320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1376256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":239,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-22T03:16:36.281284+0000","last_change":"2024-09-22T03:16:18.926944+0000","last_active":"2024-09-22T03:16:36.281284+0000","last_peered":"2024-09-22T03:16:36.281284+0000","last_clean":"2024-09-22T03:16:36.281284+0000","last_became_active":"2024-09-22T03:16:18.083437+0000","last_became_peered":"2024-09-22T03:16:18.083437+0000","last_unstale":"2024-09-22T03:16:36.281284+0000","last_undegraded":"2024-09-22T03:16:36.281284+0000","last_fullsized":"2024-09-22T03:16:36.281284+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.5","version":"106'903","reported_seq":1039,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-22T03:16:33.900283+0000","last_change":"2024-09-22T03:16:18.004962+0000","last_active":"2024-09-22T03:16:33.900283+0000","last_peered":"2024-09-22T03:16:33.900283+0000","last_clean":"2024-09-22T03:16:33.900283+0000","last_became_active":"2024-09-22T03:16:18.000028+0000","last_became_peered":"2024-09-22T03:16:18.000028+0000","last_unstale":"2024-09-22T03:16:33.900283+0000","last_undegraded":"2024-09-22T03:16:33.900283+0000","last_fullsized":"2024-09-22T03:16:33.900283+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_clean_scrub_stamp":"2024-09-22T03:15:18.897706+0000","objects_scrubbed":0,"log_size":903,"log_dups_size":0,"ondisk_log_size":903,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7405568,"num_objects":113,"num_object_clones":0,"num_object_copies":339,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":904,"num_write_kb":7232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":1966080,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":215,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-22T03:16:22.964343+0000","last_change":"2024-09-22T03:15:30.924363+0000","last_active":"2024-09-22T03:16:22.964343+0000","last_peered":"2024-09-22T03:16:22.964343+0000","last_clean":"2024-09-22T03:16:22.964343+0000","last_became_active":"2024-09-22T03:15:29.958549+0000","last_became_peered":"2024-09-22T03:15:29.958549+0000","last_unstale":"2024-09-22T03:16:22.964343+0000","last_undegraded":"2024-09-22T03:16:22.964343+0000","last_fullsized":"2024-09-22T03:16:22.964343+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"12'97","reported_seq":364,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-22T03:16:36.316682+0000","last_change":"2024-09-22T03:16:17.998607+0000","last_active":"2024-09-22T03:16:36.316682+0000","last_peered":"2024-09-22T03:16:36.316682+0000","last_clean":"2024-09-22T03:16:36.316682+0000","last_became_active":"2024-09-22T03:16:17.995046+0000","last_became_peered":"2024-09-22T03:16:17.995046+0000","last_unstale":"2024-09-22T03:16:36.316682+0000","last_undegraded":"2024-09-22T03:16:36.316682+0000","last_fullsized":"2024-09-22T03:16:36.316682+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.2","version":"106'919","reported_seq":1007,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-22T03:16:33.767173+0000","last_change":"2024-09-22T03:15:33.953595+0000","last_active":"2024-09-22T03:16:33.767173+0000","last_peered":"2024-09-22T03:16:33.767173+0000","last_clean":"2024-09-22T03:16:33.767173+0000","last_became_active":"2024-09-22T03:15:33.947435+0000","last_became_peered":"2024-09-22T03:15:33.947435+0000","last_unstale":"2024-09-22T03:16:33.767173+0000","last_undegraded":"2024-09-22T03:16:33.767173+0000","last_fullsized":"2024-09-22T03:16:33.767173+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_clean_scrub_stamp":"2024-09-22T03:15:18.897706+0000","objects_scrubbed":0,"log_size":919,"log_dups_size":0,"ondisk_log_size":919,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7536640,"num_objects":115,"num_object_clones":0,"num_object_copies":345,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":920,"num_write_kb":7360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1376256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":244,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-22T03:16:22.968835+0000","last_change":"2024-09-22T03:16:18.925553+0000","last_active":"2024-09-22T03:16:22.968835+0000","last_peered":"2024-09-22T03:16:22.968835+0000","last_clean":"2024-09-22T03:16:22.968835+0000","last_became_active":"2024-09-22T03:16:18.021400+0000","last_became_peered":"2024-09-22T03:16:18.021400+0000","last_unstale":"2024-09-22T03:16:22.968835+0000","last_undegraded":"2024-09-22T03:16:22.968835+0000","last_fullsized":"2024-09-22T03:16:22.968835+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.3","version":"106'919","reported_seq":1004,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-22T03:16:32.416964+0000","last_change":"2024-09-22T03:16:18.004775+0000","last_active":"2024-09-22T03:16:32.416964+0000","last_peered":"2024-09-22T03:16:32.416964+0000","last_clean":"2024-09-22T03:16:32.416964+0000","last_became_active":"2024-09-22T03:16:17.999612+0000","last_became_peered":"2024-09-22T03:16:17.999612+0000","last_unstale":"2024-09-22T03:16:32.416964+0000","last_undegraded":"2024-09-22T03:16:32.416964+0000","last_fullsized":"2024-09-22T03:16:32.416964+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_clean_scrub_stamp":"2024-09-22T03:15:18.897706+0000","objects_scrubbed":0,"log_size":919,"log_dups_size":0,"ondisk_log_size":919,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7536640,"num_objects":115,"num_object_clones":0,"num_object_copies":345,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":920,"num_write_kb":7360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1376256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":218,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-22T03:16:36.280703+0000","last_change":"2024-09-22T03:16:18.927373+0000","last_active":"2024-09-22T03:16:36.280703+0000","last_peered":"2024-09-22T03:16:36.280703+0000","last_clean":"2024-09-22T03:16:36.280703+0000","last_became_active":"2024-09-22T03:16:18.086190+0000","last_became_peered":"2024-09-22T03:16:18.086190+0000","last_unstale":"2024-09-22T03:16:36.280703+0000","last_undegraded":"2024-09-22T03:16:36.280703+0000","last_fullsized":"2024-09-22T03:16:36.280703+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.0","version":"106'968","reported_seq":1089,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-22T03:16:36.277367+0000","last_change":"2024-09-22T03:16:18.066854+0000","last_active":"2024-09-22T03:16:36.277367+0000","last_peered":"2024-09-22T03:16:36.277367+0000","last_clean":"2024-09-22T03:16:36.277367+0000","last_became_active":"2024-09-22T03:16:18.055858+0000","last_became_peered":"2024-09-22T03:16:18.055858+0000","last_unstale":"2024-09-22T03:16:36.277367+0000","last_undegraded":"2024-09-22T03:16:36.277367+0000","last_fullsized":"2024-09-22T03:16:36.277367+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_clean_scrub_stamp":"2024-09-22T03:15:18.897706+0000","objects_scrubbed":0,"log_size":968,"log_dups_size":0,"ondisk_log_size":968,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7929856,"num_objects":121,"num_object_clones":0,"num_object_copies":363,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":968,"num_write_kb":7744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1327104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":258,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-22T03:16:22.969675+0000","last_change":"2024-09-22T03:16:18.928041+0000","last_active":"2024-09-22T03:16:22.969675+0000","last_peered":"2024-09-22T03:16:22.969675+0000","last_clean":"2024-09-22T03:16:22.969675+0000","last_became_active":"2024-09-22T03:16:18.025488+0000","last_became_peered":"2024-09-22T03:16:18.025488+0000","last_unstale":"2024-09-22T03:16:22.969675+0000","last_undegraded":"2024-09-22T03:16:22.969675+0000","last_fullsized":"2024-09-22T03:16:22.969675+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1","version":"107'895","reported_seq":1000,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-22T03:16:36.681448+0000","last_change":"2024-09-22T03:15:33.950769+0000","last_active":"2024-09-22T03:16:36.681448+0000","last_peered":"2024-09-22T03:16:36.681448+0000","last_clean":"2024-09-22T03:16:36.681448+0000","last_became_active":"2024-09-22T03:15:33.945111+0000","last_became_peered":"2024-09-22T03:15:33.945111+0000","last_unstale":"2024-09-22T03:16:36.681448+0000","last_undegraded":"2024-09-22T03:16:36.681448+0000","last_fullsized":"2024-09-22T03:16:36.681448+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_clean_scrub_stamp":"2024-09-22T03:15:18.897706+0000","objects_scrubbed":0,"log_size":895,"log_dups_size":0,"ondisk_log_size":895,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7340032,"num_objects":112,"num_object_clones":0,"num_object_copies":336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":896,"num_write_kb":7168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":917504,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":251,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-22T03:16:22.970296+0000","last_change":"2024-09-22T03:15:30.958189+0000","last_active":"2024-09-22T03:16:22.970296+0000","last_peered":"2024-09-22T03:16:22.970296+0000","last_clean":"2024-09-22T03:16:22.970296+0000","last_became_active":"2024-09-22T03:15:29.980027+0000","last_became_peered":"2024-09-22T03:15:29.980027+0000","last_unstale":"2024-09-22T03:16:22.970296+0000","last_undegraded":"2024-09-22T03:16:22.970296+0000","last_fullsized":"2024-09-22T03:16:22.970296+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":126828544,"num_objects":1936,"num_object_clones":0,"num_object_copies":5808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15482,"num_write_kb":123856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":322,"num_bytes_recovered":21020672,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":179777536,"data_stored":379240448,"data_compressed":19492600,"data_compressed_allocated":179777536,"data_compressed_original":379240448,"omap_allocated":0,"internal_metadata":0},"log_size":15469,"ondisk_log_size":15469,"up":48,"acting":48,"num_store_stats":4},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1180736,"data_compressed":59924,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":87,"seq":373662154817,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":81144,"kb_used_data":47848,"kb_used_omap":0,"kb_used_meta":33280,"kb_avail":94290696,"statfs":{"total":96636764160,"available":96553672704,"internally_reserved":0,"allocated":48996352,"data_stored":116214482,"data_compressed":5895187,"data_compressed_allocated":48103424,"data_compressed_original":115859456,"omap_allocated":0,"internal_metadata":34078720},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":100,"seq":429496729619,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":68404,"kb_used_data":35812,"kb_used_omap":0,"kb_used_meta":32576,"kb_avail":94303436,"statfs":{"total":96636764160,"available":96566718464,"internally_reserved":0,"allocated":36671488,"data_stored":72379634,"data_compressed":3717883,"data_compressed_allocated":35758080,"data_compressed_original":72007680,"omap_allocated":0,"internal_metadata":33357824},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":53,"seq":227633266927,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":100500,"kb_used_data":50244,"kb_used_omap":0,"kb_used_meta":50240,"kb_avail":94271340,"statfs":{"total":96636764160,"available":96533852160,"internally_reserved":0,"allocated":51449856,"data_stored":101944562,"data_compressed":5244225,"data_compressed_allocated":50536448,"data_compressed_original":101572608,"omap_allocated":0,"internal_metadata":51445760},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":73,"seq":313532612728,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":98616,"kb_used_data":44264,"kb_used_omap":0,"kb_used_meta":54336,"kb_avail":94273224,"statfs":{"total":96636764160,"available":96535781376,"internally_reserved":0,"allocated":45326336,"data_stored":89708968,"data_compressed":4612250,"data_compressed_allocated":44408832,"data_compressed_original":89333760,"omap_allocated":0,"internal_metadata":55640064},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29962,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29962,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":44802048,"data_stored":89628672,"data_compressed":4627126,"data_compressed_allocated":44802048,"data_compressed_original":89628672,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":50671616,"data_stored":101351424,"data_compressed":5232452,"data_compressed_allocated":50671616,"data_compressed_original":101351424,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36057088,"data_stored":72114176,"data_compressed":3723030,"data_compressed_allocated":36057088,"data_compressed_original":72114176,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":48246784,"data_stored":116146176,"data_compressed":5909992,"data_compressed_allocated":48246784,"data_compressed_original":116146176,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-22T03:16:39.810 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-09-22T03:16:39.810 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-22T03:16:39.811 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-09-22T03:16:39.811 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-09-22T03:16:39.953 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:16:39.947+0000 7fedfceba640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:16:39.957 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:16:39.956+0000 7fedfceba640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:16:39.978 INFO:teuthology.orchestra.run.smithi153.stderr:no valid command found; 10 closest matches: 2024-09-22T03:16:39.979 INFO:teuthology.orchestra.run.smithi153.stderr:config get 2024-09-22T03:16:39.979 INFO:teuthology.orchestra.run.smithi153.stderr:config help 2024-09-22T03:16:39.979 INFO:teuthology.orchestra.run.smithi153.stderr:config set ... 2024-09-22T03:16:39.979 INFO:teuthology.orchestra.run.smithi153.stderr:config show 2024-09-22T03:16:39.979 INFO:teuthology.orchestra.run.smithi153.stderr:deep_scrub 2024-09-22T03:16:39.979 INFO:teuthology.orchestra.run.smithi153.stderr:dump_historic_ops 2024-09-22T03:16:39.979 INFO:teuthology.orchestra.run.smithi153.stderr:dump_historic_slow_ops 2024-09-22T03:16:39.979 INFO:teuthology.orchestra.run.smithi153.stderr:dump_metrics [] 2024-09-22T03:16:39.979 INFO:teuthology.orchestra.run.smithi153.stderr:dump_ops_in_flight 2024-09-22T03:16:39.979 INFO:teuthology.orchestra.run.smithi153.stderr:dump_pgstate_history 2024-09-22T03:16:39.979 INFO:teuthology.orchestra.run.smithi153.stderr:Error EINVAL: invalid command 2024-09-22T03:16:39.983 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-22T03:16:39.983 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-22T03:16:39.984 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-09-22T03:16:39.987 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 79 16 16675 16659 1.64729 2.03125 0.0287543 0.0758499 2024-09-22T03:16:40.114 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:16:40.106+0000 7f1099957640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:16:40.118 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:16:40.117+0000 7f1099957640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:16:40.158 INFO:teuthology.orchestra.run.smithi153.stderr:no valid command found; 10 closest matches: 2024-09-22T03:16:40.158 INFO:teuthology.orchestra.run.smithi153.stderr:config get 2024-09-22T03:16:40.158 INFO:teuthology.orchestra.run.smithi153.stderr:config help 2024-09-22T03:16:40.158 INFO:teuthology.orchestra.run.smithi153.stderr:config set ... 2024-09-22T03:16:40.158 INFO:teuthology.orchestra.run.smithi153.stderr:config show 2024-09-22T03:16:40.158 INFO:teuthology.orchestra.run.smithi153.stderr:deep_scrub 2024-09-22T03:16:40.158 INFO:teuthology.orchestra.run.smithi153.stderr:dump_historic_ops 2024-09-22T03:16:40.158 INFO:teuthology.orchestra.run.smithi153.stderr:dump_historic_slow_ops 2024-09-22T03:16:40.158 INFO:teuthology.orchestra.run.smithi153.stderr:dump_metrics [] 2024-09-22T03:16:40.158 INFO:teuthology.orchestra.run.smithi153.stderr:dump_ops_in_flight 2024-09-22T03:16:40.158 INFO:teuthology.orchestra.run.smithi153.stderr:dump_pgstate_history 2024-09-22T03:16:40.158 INFO:teuthology.orchestra.run.smithi153.stderr:Error EINVAL: invalid command 2024-09-22T03:16:40.162 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-22T03:16:40.162 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-22T03:16:40.163 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-09-22T03:16:40.338 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:16:40.328+0000 7fa552058640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:16:40.342 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:16:40.341+0000 7fa552058640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:16:40.356 INFO:teuthology.orchestra.run.smithi153.stderr:no valid command found; 10 closest matches: 2024-09-22T03:16:40.356 INFO:teuthology.orchestra.run.smithi153.stderr:config get 2024-09-22T03:16:40.356 INFO:teuthology.orchestra.run.smithi153.stderr:config help 2024-09-22T03:16:40.356 INFO:teuthology.orchestra.run.smithi153.stderr:config set ... 2024-09-22T03:16:40.356 INFO:teuthology.orchestra.run.smithi153.stderr:config show 2024-09-22T03:16:40.356 INFO:teuthology.orchestra.run.smithi153.stderr:deep_scrub 2024-09-22T03:16:40.356 INFO:teuthology.orchestra.run.smithi153.stderr:dump_historic_ops 2024-09-22T03:16:40.356 INFO:teuthology.orchestra.run.smithi153.stderr:dump_historic_slow_ops 2024-09-22T03:16:40.356 INFO:teuthology.orchestra.run.smithi153.stderr:dump_metrics [] 2024-09-22T03:16:40.356 INFO:teuthology.orchestra.run.smithi153.stderr:dump_ops_in_flight 2024-09-22T03:16:40.356 INFO:teuthology.orchestra.run.smithi153.stderr:dump_pgstate_history 2024-09-22T03:16:40.357 INFO:teuthology.orchestra.run.smithi153.stderr:Error EINVAL: invalid command 2024-09-22T03:16:40.359 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-22T03:16:40.359 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-22T03:16:40.360 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-09-22T03:16:40.489 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:16:40.480+0000 7fc0f06fa640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:16:40.494 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:16:40.492+0000 7fc0f06fa640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:16:40.527 INFO:teuthology.orchestra.run.smithi153.stderr:no valid command found; 10 closest matches: 2024-09-22T03:16:40.527 INFO:teuthology.orchestra.run.smithi153.stderr:config get 2024-09-22T03:16:40.527 INFO:teuthology.orchestra.run.smithi153.stderr:config help 2024-09-22T03:16:40.527 INFO:teuthology.orchestra.run.smithi153.stderr:config set ... 2024-09-22T03:16:40.528 INFO:teuthology.orchestra.run.smithi153.stderr:config show 2024-09-22T03:16:40.528 INFO:teuthology.orchestra.run.smithi153.stderr:deep_scrub 2024-09-22T03:16:40.528 INFO:teuthology.orchestra.run.smithi153.stderr:dump_historic_ops 2024-09-22T03:16:40.528 INFO:teuthology.orchestra.run.smithi153.stderr:dump_historic_slow_ops 2024-09-22T03:16:40.528 INFO:teuthology.orchestra.run.smithi153.stderr:dump_metrics [] 2024-09-22T03:16:40.528 INFO:teuthology.orchestra.run.smithi153.stderr:dump_ops_in_flight 2024-09-22T03:16:40.528 INFO:teuthology.orchestra.run.smithi153.stderr:dump_pgstate_history 2024-09-22T03:16:40.528 INFO:teuthology.orchestra.run.smithi153.stderr:Error EINVAL: invalid command 2024-09-22T03:16:40.530 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-22T03:16:40.530 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-22T03:16:40.987 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:2024-09-22T03:16:40.986598+0000 min lat: 0.00692898 max lat: 20.7098 avg lat: 0.0756161 2024-09-22T03:16:40.987 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-22T03:16:40.987 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 80 16 16938 16922 1.65238 2.05469 0.0484388 0.0756161 2024-09-22T03:16:41.505 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-22T03:16:41.633 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:16:41.629+0000 7fe420b17640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:16:41.638 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:16:41.636+0000 7fe420b17640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:16:41.987 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 81 16 17207 17191 1.65793 2.10156 0.0600633 0.0753576 2024-09-22T03:16:42.987 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 82 16 17490 17474 1.66467 2.21094 0.0581231 0.0750504 2024-09-22T03:16:43.811 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-22T03:16:43.959 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:16:43.956+0000 7f0351b97640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:16:43.965 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:16:43.963+0000 7f0351b97640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:16:43.987 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 83 16 17729 17713 1.66711 1.86719 0.0811944 0.0749541 2024-09-22T03:16:44.987 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 84 16 17940 17924 1.66688 1.64844 0.103398 0.0749517 2024-09-22T03:16:45.531 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [1, 0, 3, 2] 2024-09-22T03:16:45.531 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-22T03:16:45.531 INFO:tasks.thrashosds.thrasher:Killing osd 3, live_osds are [1, 0, 3, 2] 2024-09-22T03:16:45.532 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-09-22T03:16:45.532 INFO:teuthology.orchestra.run:waiting for 300 2024-09-22T03:16:45.987 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 85 16 18059 18043 1.65821 0.929688 0.0500601 0.0749512 2024-09-22T03:16:46.323 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-22T03:16:46.426 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:16:46.422+0000 7fd39a0a9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:16:46.429 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:16:46.426+0000 7fd39a0a9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:16:46.988 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 86 16 18059 18043 1.63893 0 - 0.0749512 2024-09-22T03:16:47.988 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 87 16 18059 18043 1.62009 0 - 0.0749512 2024-09-22T03:16:48.002 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~0s 2024-09-22T03:16:48.988 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 88 16 18059 18043 1.60168 0 - 0.0749512 2024-09-22T03:16:49.988 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 89 16 18059 18043 1.58368 0 - 0.0749512 2024-09-22T03:16:50.988 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 90 16 18059 18043 1.56608 0 - 0.0749512 2024-09-22T03:16:51.633 INFO:tasks.ceph.osd.3:Stopped 2024-09-22T03:16:51.988 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 91 16 18059 18043 1.54887 0 - 0.0749512 2024-09-22T03:16:52.988 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 92 16 18059 18043 1.53204 0 - 0.0749512 2024-09-22T03:16:53.507 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 has been restored 2024-09-22T03:16:53.989 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 93 16 18059 18043 1.51556 0 - 0.0749512 2024-09-22T03:16:54.989 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 94 16 18059 18043 1.49944 0 - 0.0749512 2024-09-22T03:16:55.989 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 95 16 18059 18043 1.48365 0 - 0.0749512 2024-09-22T03:16:56.634 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [3] live_osds: [1, 0, 2] 2024-09-22T03:16:56.634 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-22T03:16:56.634 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-22T03:16:56.752 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:16:56.746+0000 7f6dc9997640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:16:56.755 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:16:56.753+0000 7f6dc9997640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:16:56.931 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-22T03:16:56.932 INFO:teuthology.orchestra.run.smithi153.stderr:dumped all 2024-09-22T03:16:56.943 INFO:teuthology.orchestra.run.smithi153.stdout:{"pg_ready":true,"pg_map":{"version":370,"stamp":"2024-09-22T03:16:56.219963+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":147612211,"num_objects":2248,"num_object_clones":0,"num_object_copies":6744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":18086,"num_write_kb":145472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":322,"num_bytes_recovered":21020672,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18031,"ondisk_log_size":18031,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":422968,"kb_used_data":209464,"kb_used_omap":0,"kb_used_meta":213440,"kb_avail":377064392,"statfs":{"total":386547056640,"available":386113937408,"internally_reserved":0,"allocated":214491136,"data_stored":444145189,"data_compressed":22766407,"data_compressed_allocated":210731008,"data_compressed_original":442621952,"omap_allocated":0,"internal_metadata":218562560},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":3375104,"num_objects":52,"num_object_clones":0,"num_object_copies":156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":412,"num_write_kb":3296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002522"},"pg_stats":[{"pgid":"7.f","version":"110'1143","reported_seq":1237,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-22T03:16:43.895245+0000","last_change":"2024-09-22T03:15:33.972925+0000","last_active":"2024-09-22T03:16:43.895245+0000","last_peered":"2024-09-22T03:16:43.895245+0000","last_clean":"2024-09-22T03:16:43.895245+0000","last_became_active":"2024-09-22T03:15:33.967199+0000","last_became_peered":"2024-09-22T03:15:33.967199+0000","last_unstale":"2024-09-22T03:16:43.895245+0000","last_undegraded":"2024-09-22T03:16:43.895245+0000","last_fullsized":"2024-09-22T03:16:43.895245+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_clean_scrub_stamp":"2024-09-22T03:15:18.897706+0000","objects_scrubbed":0,"log_size":1143,"log_dups_size":0,"ondisk_log_size":1143,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9371648,"num_objects":143,"num_object_clones":0,"num_object_copies":429,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1144,"num_write_kb":9152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":1245184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.e","version":"110'1175","reported_seq":1261,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-22T03:16:43.162149+0000","last_change":"2024-09-22T03:16:18.028856+0000","last_active":"2024-09-22T03:16:43.162149+0000","last_peered":"2024-09-22T03:16:43.162149+0000","last_clean":"2024-09-22T03:16:43.162149+0000","last_became_active":"2024-09-22T03:16:18.021495+0000","last_became_peered":"2024-09-22T03:16:18.021495+0000","last_unstale":"2024-09-22T03:16:43.162149+0000","last_undegraded":"2024-09-22T03:16:43.162149+0000","last_fullsized":"2024-09-22T03:16:43.162149+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_clean_scrub_stamp":"2024-09-22T03:15:18.897706+0000","objects_scrubbed":0,"log_size":1175,"log_dups_size":0,"ondisk_log_size":1175,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9633792,"num_objects":147,"num_object_clones":0,"num_object_copies":441,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1176,"num_write_kb":9408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.d","version":"110'1199","reported_seq":1341,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-22T03:16:45.328734+0000","last_change":"2024-09-22T03:16:18.042408+0000","last_active":"2024-09-22T03:16:45.328734+0000","last_peered":"2024-09-22T03:16:45.328734+0000","last_clean":"2024-09-22T03:16:45.328734+0000","last_became_active":"2024-09-22T03:16:18.029899+0000","last_became_peered":"2024-09-22T03:16:18.029899+0000","last_unstale":"2024-09-22T03:16:45.328734+0000","last_undegraded":"2024-09-22T03:16:45.328734+0000","last_fullsized":"2024-09-22T03:16:45.328734+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_clean_scrub_stamp":"2024-09-22T03:15:18.897706+0000","objects_scrubbed":0,"log_size":1199,"log_dups_size":0,"ondisk_log_size":1199,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9830400,"num_objects":150,"num_object_clones":0,"num_object_copies":450,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1200,"num_write_kb":9600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":1835008,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.c","version":"110'1255","reported_seq":1373,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-22T03:16:45.272727+0000","last_change":"2024-09-22T03:16:18.031863+0000","last_active":"2024-09-22T03:16:45.272727+0000","last_peered":"2024-09-22T03:16:45.272727+0000","last_clean":"2024-09-22T03:16:45.272727+0000","last_became_active":"2024-09-22T03:16:18.022365+0000","last_became_peered":"2024-09-22T03:16:18.022365+0000","last_unstale":"2024-09-22T03:16:45.272727+0000","last_undegraded":"2024-09-22T03:16:45.272727+0000","last_fullsized":"2024-09-22T03:16:45.272727+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_clean_scrub_stamp":"2024-09-22T03:15:18.897706+0000","objects_scrubbed":0,"log_size":1255,"log_dups_size":0,"ondisk_log_size":1255,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10289152,"num_objects":157,"num_object_clones":0,"num_object_copies":471,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1256,"num_write_kb":10048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":1048576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.b","version":"110'1055","reported_seq":1128,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-22T03:16:42.584706+0000","last_change":"2024-09-22T03:15:34.925809+0000","last_active":"2024-09-22T03:16:42.584706+0000","last_peered":"2024-09-22T03:16:42.584706+0000","last_clean":"2024-09-22T03:16:42.584706+0000","last_became_active":"2024-09-22T03:15:34.920855+0000","last_became_peered":"2024-09-22T03:15:34.920855+0000","last_unstale":"2024-09-22T03:16:42.584706+0000","last_undegraded":"2024-09-22T03:16:42.584706+0000","last_fullsized":"2024-09-22T03:16:42.584706+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_clean_scrub_stamp":"2024-09-22T03:15:18.897706+0000","objects_scrubbed":0,"log_size":1055,"log_dups_size":0,"ondisk_log_size":1055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8650752,"num_objects":132,"num_object_clones":0,"num_object_copies":396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1056,"num_write_kb":8448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":2031616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"110'1007","reported_seq":1084,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-22T03:16:43.183216+0000","last_change":"2024-09-22T03:16:18.015447+0000","last_active":"2024-09-22T03:16:43.183216+0000","last_peered":"2024-09-22T03:16:43.183216+0000","last_clean":"2024-09-22T03:16:43.183216+0000","last_became_active":"2024-09-22T03:16:18.012158+0000","last_became_peered":"2024-09-22T03:16:18.012158+0000","last_unstale":"2024-09-22T03:16:43.183216+0000","last_undegraded":"2024-09-22T03:16:43.183216+0000","last_fullsized":"2024-09-22T03:16:43.183216+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_clean_scrub_stamp":"2024-09-22T03:15:18.897706+0000","objects_scrubbed":0,"log_size":1007,"log_dups_size":0,"ondisk_log_size":1007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8257536,"num_objects":126,"num_object_clones":0,"num_object_copies":378,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1008,"num_write_kb":8064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":1507328,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.9","version":"110'999","reported_seq":1121,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-22T03:16:44.775958+0000","last_change":"2024-09-22T03:16:18.095024+0000","last_active":"2024-09-22T03:16:44.775958+0000","last_peered":"2024-09-22T03:16:44.775958+0000","last_clean":"2024-09-22T03:16:44.775958+0000","last_became_active":"2024-09-22T03:16:18.088681+0000","last_became_peered":"2024-09-22T03:16:18.088681+0000","last_unstale":"2024-09-22T03:16:44.775958+0000","last_undegraded":"2024-09-22T03:16:44.775958+0000","last_fullsized":"2024-09-22T03:16:44.775958+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_clean_scrub_stamp":"2024-09-22T03:15:18.897706+0000","objects_scrubbed":0,"log_size":999,"log_dups_size":0,"ondisk_log_size":999,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8192000,"num_objects":125,"num_object_clones":0,"num_object_copies":375,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1000,"num_write_kb":8000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":1179648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.8","version":"110'1103","reported_seq":1235,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-22T03:16:44.830154+0000","last_change":"2024-09-22T03:15:33.946927+0000","last_active":"2024-09-22T03:16:44.830154+0000","last_peered":"2024-09-22T03:16:44.830154+0000","last_clean":"2024-09-22T03:16:44.830154+0000","last_became_active":"2024-09-22T03:15:33.940267+0000","last_became_peered":"2024-09-22T03:15:33.940267+0000","last_unstale":"2024-09-22T03:16:44.830154+0000","last_undegraded":"2024-09-22T03:16:44.830154+0000","last_fullsized":"2024-09-22T03:16:44.830154+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_clean_scrub_stamp":"2024-09-22T03:15:18.897706+0000","objects_scrubbed":0,"log_size":1103,"log_dups_size":0,"ondisk_log_size":1103,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9043968,"num_objects":138,"num_object_clones":0,"num_object_copies":414,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1104,"num_write_kb":8832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":1507328,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.6","version":"110'1223","reported_seq":1328,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-22T03:16:43.575597+0000","last_change":"2024-09-22T03:15:32.999617+0000","last_active":"2024-09-22T03:16:43.575597+0000","last_peered":"2024-09-22T03:16:43.575597+0000","last_clean":"2024-09-22T03:16:43.575597+0000","last_became_active":"2024-09-22T03:15:32.993508+0000","last_became_peered":"2024-09-22T03:15:32.993508+0000","last_unstale":"2024-09-22T03:16:43.575597+0000","last_undegraded":"2024-09-22T03:16:43.575597+0000","last_fullsized":"2024-09-22T03:16:43.575597+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_clean_scrub_stamp":"2024-09-22T03:15:18.897706+0000","objects_scrubbed":0,"log_size":1223,"log_dups_size":0,"ondisk_log_size":1223,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10027008,"num_objects":153,"num_object_clones":0,"num_object_copies":459,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1224,"num_write_kb":9792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":491520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":168,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-22T03:16:38.931930+0000","last_change":"2024-09-22T03:16:18.969989+0000","last_active":"2024-09-22T03:16:38.931930+0000","last_peered":"2024-09-22T03:16:38.931930+0000","last_clean":"2024-09-22T03:16:38.931930+0000","last_became_active":"2024-09-22T03:16:18.003684+0000","last_became_peered":"2024-09-22T03:16:18.003684+0000","last_unstale":"2024-09-22T03:16:38.931930+0000","last_undegraded":"2024-09-22T03:16:38.931930+0000","last_fullsized":"2024-09-22T03:16:38.931930+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.7","version":"110'1287","reported_seq":1429,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-22T03:16:45.286966+0000","last_change":"2024-09-22T03:15:33.949109+0000","last_active":"2024-09-22T03:16:45.286966+0000","last_peered":"2024-09-22T03:16:45.286966+0000","last_clean":"2024-09-22T03:16:45.286966+0000","last_became_active":"2024-09-22T03:15:33.943481+0000","last_became_peered":"2024-09-22T03:15:33.943481+0000","last_unstale":"2024-09-22T03:16:45.286966+0000","last_undegraded":"2024-09-22T03:16:45.286966+0000","last_fullsized":"2024-09-22T03:16:45.286966+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_clean_scrub_stamp":"2024-09-22T03:15:18.897706+0000","objects_scrubbed":0,"log_size":1287,"log_dups_size":0,"ondisk_log_size":1287,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10551296,"num_objects":161,"num_object_clones":0,"num_object_copies":483,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1288,"num_write_kb":10304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":1835008,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":251,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-22T03:16:38.928354+0000","last_change":"2024-09-22T03:16:18.925677+0000","last_active":"2024-09-22T03:16:38.928354+0000","last_peered":"2024-09-22T03:16:38.928354+0000","last_clean":"2024-09-22T03:16:38.928354+0000","last_became_active":"2024-09-22T03:16:17.986660+0000","last_became_peered":"2024-09-22T03:16:17.986660+0000","last_unstale":"2024-09-22T03:16:38.928354+0000","last_undegraded":"2024-09-22T03:16:38.928354+0000","last_fullsized":"2024-09-22T03:16:38.928354+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.4","version":"110'1151","reported_seq":1279,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-22T03:16:45.198422+0000","last_change":"2024-09-22T03:16:18.072690+0000","last_active":"2024-09-22T03:16:45.198422+0000","last_peered":"2024-09-22T03:16:45.198422+0000","last_clean":"2024-09-22T03:16:45.198422+0000","last_became_active":"2024-09-22T03:16:18.063063+0000","last_became_peered":"2024-09-22T03:16:18.063063+0000","last_unstale":"2024-09-22T03:16:45.198422+0000","last_undegraded":"2024-09-22T03:16:45.198422+0000","last_fullsized":"2024-09-22T03:16:45.198422+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_clean_scrub_stamp":"2024-09-22T03:15:18.897706+0000","objects_scrubbed":0,"log_size":1151,"log_dups_size":0,"ondisk_log_size":1151,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9437184,"num_objects":144,"num_object_clones":0,"num_object_copies":432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1152,"num_write_kb":9216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1376256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":248,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-22T03:16:38.920473+0000","last_change":"2024-09-22T03:16:18.926944+0000","last_active":"2024-09-22T03:16:38.920473+0000","last_peered":"2024-09-22T03:16:38.920473+0000","last_clean":"2024-09-22T03:16:38.920473+0000","last_became_active":"2024-09-22T03:16:18.083437+0000","last_became_peered":"2024-09-22T03:16:18.083437+0000","last_unstale":"2024-09-22T03:16:38.920473+0000","last_undegraded":"2024-09-22T03:16:38.920473+0000","last_fullsized":"2024-09-22T03:16:38.920473+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.5","version":"110'1034","reported_seq":1182,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-22T03:16:45.537744+0000","last_change":"2024-09-22T03:16:18.004962+0000","last_active":"2024-09-22T03:16:45.537744+0000","last_peered":"2024-09-22T03:16:45.537744+0000","last_clean":"2024-09-22T03:16:45.537744+0000","last_became_active":"2024-09-22T03:16:18.000028+0000","last_became_peered":"2024-09-22T03:16:18.000028+0000","last_unstale":"2024-09-22T03:16:45.537744+0000","last_undegraded":"2024-09-22T03:16:45.537744+0000","last_fullsized":"2024-09-22T03:16:45.537744+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_clean_scrub_stamp":"2024-09-22T03:15:18.897706+0000","objects_scrubbed":0,"log_size":1034,"log_dups_size":0,"ondisk_log_size":1034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8478720,"num_objects":130,"num_object_clones":0,"num_object_copies":390,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1035,"num_write_kb":8280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":1966080,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":227,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-22T03:16:38.932586+0000","last_change":"2024-09-22T03:15:30.924363+0000","last_active":"2024-09-22T03:16:38.932586+0000","last_peered":"2024-09-22T03:16:38.932586+0000","last_clean":"2024-09-22T03:16:38.932586+0000","last_became_active":"2024-09-22T03:15:29.958549+0000","last_became_peered":"2024-09-22T03:15:29.958549+0000","last_unstale":"2024-09-22T03:16:38.932586+0000","last_undegraded":"2024-09-22T03:16:38.932586+0000","last_fullsized":"2024-09-22T03:16:38.932586+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"12'97","reported_seq":373,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-22T03:16:38.931051+0000","last_change":"2024-09-22T03:16:17.998607+0000","last_active":"2024-09-22T03:16:38.931051+0000","last_peered":"2024-09-22T03:16:38.931051+0000","last_clean":"2024-09-22T03:16:38.931051+0000","last_became_active":"2024-09-22T03:16:17.995046+0000","last_became_peered":"2024-09-22T03:16:17.995046+0000","last_unstale":"2024-09-22T03:16:38.931051+0000","last_undegraded":"2024-09-22T03:16:38.931051+0000","last_fullsized":"2024-09-22T03:16:38.931051+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.2","version":"110'1047","reported_seq":1147,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-22T03:16:42.656280+0000","last_change":"2024-09-22T03:15:33.953595+0000","last_active":"2024-09-22T03:16:42.656280+0000","last_peered":"2024-09-22T03:16:42.656280+0000","last_clean":"2024-09-22T03:16:42.656280+0000","last_became_active":"2024-09-22T03:15:33.947435+0000","last_became_peered":"2024-09-22T03:15:33.947435+0000","last_unstale":"2024-09-22T03:16:42.656280+0000","last_undegraded":"2024-09-22T03:16:42.656280+0000","last_fullsized":"2024-09-22T03:16:42.656280+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_clean_scrub_stamp":"2024-09-22T03:15:18.897706+0000","objects_scrubbed":0,"log_size":1047,"log_dups_size":0,"ondisk_log_size":1047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8585216,"num_objects":131,"num_object_clones":0,"num_object_copies":393,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1048,"num_write_kb":8384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1376256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":256,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-22T03:16:38.927288+0000","last_change":"2024-09-22T03:16:18.925553+0000","last_active":"2024-09-22T03:16:38.927288+0000","last_peered":"2024-09-22T03:16:38.927288+0000","last_clean":"2024-09-22T03:16:38.927288+0000","last_became_active":"2024-09-22T03:16:18.021400+0000","last_became_peered":"2024-09-22T03:16:18.021400+0000","last_unstale":"2024-09-22T03:16:38.927288+0000","last_undegraded":"2024-09-22T03:16:38.927288+0000","last_fullsized":"2024-09-22T03:16:38.927288+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.3","version":"110'1063","reported_seq":1160,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-22T03:16:43.409760+0000","last_change":"2024-09-22T03:16:18.004775+0000","last_active":"2024-09-22T03:16:43.409760+0000","last_peered":"2024-09-22T03:16:43.409760+0000","last_clean":"2024-09-22T03:16:43.409760+0000","last_became_active":"2024-09-22T03:16:17.999612+0000","last_became_peered":"2024-09-22T03:16:17.999612+0000","last_unstale":"2024-09-22T03:16:43.409760+0000","last_undegraded":"2024-09-22T03:16:43.409760+0000","last_fullsized":"2024-09-22T03:16:43.409760+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_clean_scrub_stamp":"2024-09-22T03:15:18.897706+0000","objects_scrubbed":0,"log_size":1063,"log_dups_size":0,"ondisk_log_size":1063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8716288,"num_objects":133,"num_object_clones":0,"num_object_copies":399,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1064,"num_write_kb":8512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1376256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":227,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-22T03:16:38.920680+0000","last_change":"2024-09-22T03:16:18.927373+0000","last_active":"2024-09-22T03:16:38.920680+0000","last_peered":"2024-09-22T03:16:38.920680+0000","last_clean":"2024-09-22T03:16:38.920680+0000","last_became_active":"2024-09-22T03:16:18.086190+0000","last_became_peered":"2024-09-22T03:16:18.086190+0000","last_unstale":"2024-09-22T03:16:38.920680+0000","last_undegraded":"2024-09-22T03:16:38.920680+0000","last_fullsized":"2024-09-22T03:16:38.920680+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.0","version":"110'1159","reported_seq":1290,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-22T03:16:45.101963+0000","last_change":"2024-09-22T03:16:18.066854+0000","last_active":"2024-09-22T03:16:45.101963+0000","last_peered":"2024-09-22T03:16:45.101963+0000","last_clean":"2024-09-22T03:16:45.101963+0000","last_became_active":"2024-09-22T03:16:18.055858+0000","last_became_peered":"2024-09-22T03:16:18.055858+0000","last_unstale":"2024-09-22T03:16:45.101963+0000","last_undegraded":"2024-09-22T03:16:45.101963+0000","last_fullsized":"2024-09-22T03:16:45.101963+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_clean_scrub_stamp":"2024-09-22T03:15:18.897706+0000","objects_scrubbed":0,"log_size":1159,"log_dups_size":0,"ondisk_log_size":1159,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9502720,"num_objects":145,"num_object_clones":0,"num_object_copies":435,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1160,"num_write_kb":9280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1327104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":270,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-22T03:16:38.927396+0000","last_change":"2024-09-22T03:16:18.928041+0000","last_active":"2024-09-22T03:16:38.927396+0000","last_peered":"2024-09-22T03:16:38.927396+0000","last_clean":"2024-09-22T03:16:38.927396+0000","last_became_active":"2024-09-22T03:16:18.025488+0000","last_became_peered":"2024-09-22T03:16:18.025488+0000","last_unstale":"2024-09-22T03:16:38.927396+0000","last_undegraded":"2024-09-22T03:16:38.927396+0000","last_fullsized":"2024-09-22T03:16:38.927396+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1","version":"110'1031","reported_seq":1145,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-22T03:16:44.305650+0000","last_change":"2024-09-22T03:15:33.950769+0000","last_active":"2024-09-22T03:16:44.305650+0000","last_peered":"2024-09-22T03:16:44.305650+0000","last_clean":"2024-09-22T03:16:44.305650+0000","last_became_active":"2024-09-22T03:15:33.945111+0000","last_became_peered":"2024-09-22T03:15:33.945111+0000","last_unstale":"2024-09-22T03:16:44.305650+0000","last_undegraded":"2024-09-22T03:16:44.305650+0000","last_fullsized":"2024-09-22T03:16:44.305650+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:15:18.897706+0000","last_clean_scrub_stamp":"2024-09-22T03:15:18.897706+0000","objects_scrubbed":0,"log_size":1031,"log_dups_size":0,"ondisk_log_size":1031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8454144,"num_objects":129,"num_object_clones":0,"num_object_copies":387,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1032,"num_write_kb":8256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":917504,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":263,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-22T03:16:38.927750+0000","last_change":"2024-09-22T03:15:30.958189+0000","last_active":"2024-09-22T03:16:38.927750+0000","last_peered":"2024-09-22T03:16:38.927750+0000","last_clean":"2024-09-22T03:16:38.927750+0000","last_became_active":"2024-09-22T03:15:29.980027+0000","last_became_peered":"2024-09-22T03:15:29.980027+0000","last_unstale":"2024-09-22T03:16:38.927750+0000","last_undegraded":"2024-09-22T03:16:38.927750+0000","last_fullsized":"2024-09-22T03:16:38.927750+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":147021824,"num_objects":2244,"num_object_clones":0,"num_object_copies":6732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17947,"num_write_kb":143576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":322,"num_bytes_recovered":21020672,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":210735104,"data_stored":441155584,"data_compressed":22689634,"data_compressed_allocated":210735104,"data_compressed_original":441155584,"omap_allocated":0,"internal_metadata":0},"log_size":17931,"ondisk_log_size":17931,"up":48,"acting":48,"num_store_stats":4},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1180736,"data_compressed":59924,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":87,"seq":373662154827,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":117136,"kb_used_data":57472,"kb_used_omap":0,"kb_used_meta":59648,"kb_avail":94254704,"statfs":{"total":96636764160,"available":96516816896,"internally_reserved":0,"allocated":58851328,"data_stored":135871990,"data_compressed":6909541,"data_compressed_allocated":57925632,"data_compressed_original":135503872,"omap_allocated":0,"internal_metadata":61079552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":100,"seq":429496729634,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":77984,"kb_used_data":41232,"kb_used_omap":0,"kb_used_meta":36736,"kb_avail":94293856,"statfs":{"total":96636764160,"available":96556908544,"internally_reserved":0,"allocated":42221568,"data_stored":83427350,"data_compressed":4287664,"data_compressed_allocated":41275392,"data_compressed_original":83042304,"omap_allocated":0,"internal_metadata":37617664},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":53,"seq":227633266947,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":118784,"kb_used_data":60208,"kb_used_omap":0,"kb_used_meta":58560,"kb_avail":94253056,"statfs":{"total":96636764160,"available":96515129344,"internally_reserved":0,"allocated":61652992,"data_stored":122298390,"data_compressed":6294534,"data_compressed_allocated":60706816,"data_compressed_original":121913344,"omap_allocated":0,"internal_metadata":59965440},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":73,"seq":313532612743,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":109064,"kb_used_data":50552,"kb_used_omap":0,"kb_used_meta":58496,"kb_avail":94262776,"statfs":{"total":96636764160,"available":96525082624,"internally_reserved":0,"allocated":51765248,"data_stored":102547459,"data_compressed":5274668,"data_compressed_allocated":50823168,"data_compressed_original":102162432,"omap_allocated":0,"internal_metadata":59899904},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29962,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29962,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":50782208,"data_stored":101588992,"data_compressed":5244706,"data_compressed_allocated":50782208,"data_compressed_original":101588992,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":60665856,"data_stored":121339904,"data_compressed":6264572,"data_compressed_allocated":60665856,"data_compressed_original":121339904,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":41234432,"data_stored":82468864,"data_compressed":4257702,"data_compressed_allocated":41234432,"data_compressed_original":82468864,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":58052608,"data_stored":135757824,"data_compressed":6922654,"data_compressed_allocated":58052608,"data_compressed_original":135757824,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-22T03:16:56.989 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 96 16 18059 18043 1.4682 0 - 0.0749512 2024-09-22T03:16:57.989 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 97 16 18059 18043 1.45306 0 - 0.0749512 2024-09-22T03:16:58.989 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 98 16 18059 18043 1.43823 0 - 0.0749512 2024-09-22T03:16:59.989 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 99 16 18059 18043 1.42371 0 - 0.0749512 2024-09-22T03:17:00.989 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:2024-09-22T03:17:00.989078+0000 min lat: 0.00692898 max lat: 20.7098 avg lat: 0.0749512 2024-09-22T03:17:00.990 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-22T03:17:00.990 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 100 16 18059 18043 1.40947 0 - 0.0749512 2024-09-22T03:17:01.946 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [3] live_osds: [1, 0, 2] 2024-09-22T03:17:01.946 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-22T03:17:01.946 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-09-22T03:17:01.946 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-09-22T03:17:01.990 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 101 16 18059 18043 1.39551 0 - 0.0749512 2024-09-22T03:17:02.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:17:02.059+0000 7f4f7b2b2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:17:02.068 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:17:02.067+0000 7f4f7b2b2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:17:02.080 INFO:teuthology.orchestra.run.smithi153.stderr:no valid command found; 10 closest matches: 2024-09-22T03:17:02.080 INFO:teuthology.orchestra.run.smithi153.stderr:config get 2024-09-22T03:17:02.080 INFO:teuthology.orchestra.run.smithi153.stderr:config help 2024-09-22T03:17:02.081 INFO:teuthology.orchestra.run.smithi153.stderr:config set ... 2024-09-22T03:17:02.081 INFO:teuthology.orchestra.run.smithi153.stderr:config show 2024-09-22T03:17:02.081 INFO:teuthology.orchestra.run.smithi153.stderr:deep_scrub 2024-09-22T03:17:02.081 INFO:teuthology.orchestra.run.smithi153.stderr:dump_historic_ops 2024-09-22T03:17:02.081 INFO:teuthology.orchestra.run.smithi153.stderr:dump_historic_slow_ops 2024-09-22T03:17:02.081 INFO:teuthology.orchestra.run.smithi153.stderr:dump_metrics [] 2024-09-22T03:17:02.081 INFO:teuthology.orchestra.run.smithi153.stderr:dump_ops_in_flight 2024-09-22T03:17:02.081 INFO:teuthology.orchestra.run.smithi153.stderr:dump_pgstate_history 2024-09-22T03:17:02.081 INFO:teuthology.orchestra.run.smithi153.stderr:Error EINVAL: invalid command 2024-09-22T03:17:02.083 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-22T03:17:02.083 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-22T03:17:02.084 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-09-22T03:17:02.230 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:17:02.224+0000 7ff8c3c58640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:17:02.233 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:17:02.232+0000 7ff8c3c58640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:17:02.245 INFO:teuthology.orchestra.run.smithi153.stderr:no valid command found; 10 closest matches: 2024-09-22T03:17:02.245 INFO:teuthology.orchestra.run.smithi153.stderr:config get 2024-09-22T03:17:02.245 INFO:teuthology.orchestra.run.smithi153.stderr:config help 2024-09-22T03:17:02.245 INFO:teuthology.orchestra.run.smithi153.stderr:config set ... 2024-09-22T03:17:02.245 INFO:teuthology.orchestra.run.smithi153.stderr:config show 2024-09-22T03:17:02.245 INFO:teuthology.orchestra.run.smithi153.stderr:deep_scrub 2024-09-22T03:17:02.246 INFO:teuthology.orchestra.run.smithi153.stderr:dump_historic_ops 2024-09-22T03:17:02.246 INFO:teuthology.orchestra.run.smithi153.stderr:dump_historic_slow_ops 2024-09-22T03:17:02.246 INFO:teuthology.orchestra.run.smithi153.stderr:dump_metrics [] 2024-09-22T03:17:02.246 INFO:teuthology.orchestra.run.smithi153.stderr:dump_ops_in_flight 2024-09-22T03:17:02.246 INFO:teuthology.orchestra.run.smithi153.stderr:dump_pgstate_history 2024-09-22T03:17:02.246 INFO:teuthology.orchestra.run.smithi153.stderr:Error EINVAL: invalid command 2024-09-22T03:17:02.248 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-22T03:17:02.248 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-22T03:17:02.248 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-09-22T03:17:02.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:17:02.347+0000 7f292e86c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:17:02.356 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:17:02.355+0000 7f292e86c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:17:02.369 INFO:teuthology.orchestra.run.smithi153.stderr:no valid command found; 10 closest matches: 2024-09-22T03:17:02.369 INFO:teuthology.orchestra.run.smithi153.stderr:config get 2024-09-22T03:17:02.369 INFO:teuthology.orchestra.run.smithi153.stderr:config help 2024-09-22T03:17:02.370 INFO:teuthology.orchestra.run.smithi153.stderr:config set ... 2024-09-22T03:17:02.370 INFO:teuthology.orchestra.run.smithi153.stderr:config show 2024-09-22T03:17:02.370 INFO:teuthology.orchestra.run.smithi153.stderr:deep_scrub 2024-09-22T03:17:02.370 INFO:teuthology.orchestra.run.smithi153.stderr:dump_historic_ops 2024-09-22T03:17:02.370 INFO:teuthology.orchestra.run.smithi153.stderr:dump_historic_slow_ops 2024-09-22T03:17:02.370 INFO:teuthology.orchestra.run.smithi153.stderr:dump_metrics [] 2024-09-22T03:17:02.370 INFO:teuthology.orchestra.run.smithi153.stderr:dump_ops_in_flight 2024-09-22T03:17:02.370 INFO:teuthology.orchestra.run.smithi153.stderr:dump_pgstate_history 2024-09-22T03:17:02.370 INFO:teuthology.orchestra.run.smithi153.stderr:Error EINVAL: invalid command 2024-09-22T03:17:02.372 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-22T03:17:02.372 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-22T03:17:02.372 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-09-22T03:17:02.476 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:17:02.470+0000 7f25ae797640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:17:02.479 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:17:02.478+0000 7f25ae797640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:17:02.990 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 102 16 18059 18043 1.38183 0 - 0.0749512 2024-09-22T03:17:03.990 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 103 16 18059 18043 1.36842 0 - 0.0749512 2024-09-22T03:17:04.990 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 104 16 18059 18043 1.35526 0 - 0.0749512 2024-09-22T03:17:05.990 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 105 16 18059 18043 1.34235 0 - 0.0749512 2024-09-22T03:17:06.990 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 106 16 18059 18043 1.32969 0 - 0.0749512 2024-09-22T03:17:07.990 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 107 16 18059 18043 1.31726 0 - 0.0749512 2024-09-22T03:17:08.990 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 108 16 18059 18043 1.30506 0 - 0.0749512 2024-09-22T03:17:09.991 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 109 16 18059 18043 1.29309 0 - 0.0749512 2024-09-22T03:17:10.991 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 110 16 18059 18043 1.28133 0 - 0.0749512 2024-09-22T03:17:11.945 INFO:teuthology.orchestra.run.smithi153.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-09-22T03:17:11.946 INFO:teuthology.orchestra.run.smithi153.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-09-22T03:17:11.950 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-22T03:17:11.950 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-22T03:17:11.951 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-22T03:17:11.951 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-22T03:17:11.991 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 111 16 18059 18043 1.26979 0 - 0.0749512 2024-09-22T03:17:12.991 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 112 16 18059 18043 1.25845 0 - 0.0749512 2024-09-22T03:17:13.952 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-22T03:17:13.991 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:Total time run: 112.139 2024-09-22T03:17:13.991 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:Total writes made: 18059 2024-09-22T03:17:13.991 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:Write size: 8192 2024-09-22T03:17:13.991 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:Object size: 65536 2024-09-22T03:17:13.991 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:Bandwidth (MB/sec): 1.25813 2024-09-22T03:17:13.991 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:Stddev Bandwidth: 1.18889 2024-09-22T03:17:13.992 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:Max bandwidth (MB/sec): 3.11719 2024-09-22T03:17:13.992 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:Min bandwidth (MB/sec): 0 2024-09-22T03:17:13.992 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:Average IOPS: 161 2024-09-22T03:17:13.992 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:Stddev IOPS: 152.178 2024-09-22T03:17:13.992 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:Max IOPS: 399 2024-09-22T03:17:13.992 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:Min IOPS: 0 2024-09-22T03:17:13.992 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:Average Latency(s): 0.0992965 2024-09-22T03:17:13.992 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:Stddev Latency(s): 1.02541 2024-09-22T03:17:13.992 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:Max latency(s): 27.6065 2024-09-22T03:17:13.992 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:Min latency(s): 0.00692898 2024-09-22T03:17:14.010 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:Cleaning up (deleting benchmark objects) 2024-09-22T03:17:14.099 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:17:14.095+0000 7f3968c6c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:17:14.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:17:14.102+0000 7f3968c6c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:17:14.303 INFO:teuthology.orchestra.run.smithi153.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-09-22T03:17:14.307 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-22T03:17:14.308 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-22T03:17:16.309 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-22T03:17:16.451 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:17:16.448+0000 7f0e3ec2c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:17:16.456 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:17:16.455+0000 7f0e3ec2c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:17:16.669 INFO:teuthology.orchestra.run.smithi153.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-09-22T03:17:16.673 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-22T03:17:16.674 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-22T03:17:16.951 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [3] live_osds: [1, 0, 2] 2024-09-22T03:17:16.952 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-22T03:17:16.952 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-09-22T03:17:16.952 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-09-22T03:17:17.112 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:17:17.102+0000 7fb301357640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:17:17.117 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:17:17.115+0000 7fb301357640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:17:17.140 INFO:teuthology.orchestra.run.smithi153.stderr:no valid command found; 10 closest matches: 2024-09-22T03:17:17.140 INFO:teuthology.orchestra.run.smithi153.stderr:config get 2024-09-22T03:17:17.141 INFO:teuthology.orchestra.run.smithi153.stderr:config help 2024-09-22T03:17:17.141 INFO:teuthology.orchestra.run.smithi153.stderr:config set ... 2024-09-22T03:17:17.141 INFO:teuthology.orchestra.run.smithi153.stderr:config show 2024-09-22T03:17:17.141 INFO:teuthology.orchestra.run.smithi153.stderr:deep_scrub 2024-09-22T03:17:17.141 INFO:teuthology.orchestra.run.smithi153.stderr:dump_historic_ops 2024-09-22T03:17:17.141 INFO:teuthology.orchestra.run.smithi153.stderr:dump_historic_slow_ops 2024-09-22T03:17:17.141 INFO:teuthology.orchestra.run.smithi153.stderr:dump_metrics [] 2024-09-22T03:17:17.141 INFO:teuthology.orchestra.run.smithi153.stderr:dump_ops_in_flight 2024-09-22T03:17:17.141 INFO:teuthology.orchestra.run.smithi153.stderr:dump_pgstate_history 2024-09-22T03:17:17.141 INFO:teuthology.orchestra.run.smithi153.stderr:Error EINVAL: invalid command 2024-09-22T03:17:17.144 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-22T03:17:17.145 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-22T03:17:17.145 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-09-22T03:17:17.275 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:17:17.268+0000 7f690ef57640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:17:17.281 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:17:17.279+0000 7f690ef57640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:17:17.427 INFO:teuthology.orchestra.run.smithi153.stderr:no valid command found; 10 closest matches: 2024-09-22T03:17:17.427 INFO:teuthology.orchestra.run.smithi153.stderr:config get 2024-09-22T03:17:17.427 INFO:teuthology.orchestra.run.smithi153.stderr:config help 2024-09-22T03:17:17.427 INFO:teuthology.orchestra.run.smithi153.stderr:config set ... 2024-09-22T03:17:17.427 INFO:teuthology.orchestra.run.smithi153.stderr:config show 2024-09-22T03:17:17.427 INFO:teuthology.orchestra.run.smithi153.stderr:deep_scrub 2024-09-22T03:17:17.427 INFO:teuthology.orchestra.run.smithi153.stderr:dump_historic_ops 2024-09-22T03:17:17.427 INFO:teuthology.orchestra.run.smithi153.stderr:dump_historic_slow_ops 2024-09-22T03:17:17.427 INFO:teuthology.orchestra.run.smithi153.stderr:dump_metrics [] 2024-09-22T03:17:17.427 INFO:teuthology.orchestra.run.smithi153.stderr:dump_ops_in_flight 2024-09-22T03:17:17.427 INFO:teuthology.orchestra.run.smithi153.stderr:dump_pgstate_history 2024-09-22T03:17:17.427 INFO:teuthology.orchestra.run.smithi153.stderr:Error EINVAL: invalid command 2024-09-22T03:17:17.429 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-22T03:17:17.430 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-22T03:17:17.430 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-09-22T03:17:17.575 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:17:17.566+0000 7f738a86c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:17:17.581 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:17:17.579+0000 7f738a86c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:17:17.653 INFO:teuthology.orchestra.run.smithi153.stderr:no valid command found; 10 closest matches: 2024-09-22T03:17:17.653 INFO:teuthology.orchestra.run.smithi153.stderr:config get 2024-09-22T03:17:17.653 INFO:teuthology.orchestra.run.smithi153.stderr:config help 2024-09-22T03:17:17.653 INFO:teuthology.orchestra.run.smithi153.stderr:config set ... 2024-09-22T03:17:17.653 INFO:teuthology.orchestra.run.smithi153.stderr:config show 2024-09-22T03:17:17.653 INFO:teuthology.orchestra.run.smithi153.stderr:deep_scrub 2024-09-22T03:17:17.654 INFO:teuthology.orchestra.run.smithi153.stderr:dump_historic_ops 2024-09-22T03:17:17.654 INFO:teuthology.orchestra.run.smithi153.stderr:dump_historic_slow_ops 2024-09-22T03:17:17.654 INFO:teuthology.orchestra.run.smithi153.stderr:dump_metrics [] 2024-09-22T03:17:17.654 INFO:teuthology.orchestra.run.smithi153.stderr:dump_ops_in_flight 2024-09-22T03:17:17.654 INFO:teuthology.orchestra.run.smithi153.stderr:dump_pgstate_history 2024-09-22T03:17:17.654 INFO:teuthology.orchestra.run.smithi153.stderr:Error EINVAL: invalid command 2024-09-22T03:17:17.657 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-22T03:17:17.657 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-22T03:17:17.658 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-09-22T03:17:17.817 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:17:17.808+0000 7f90e7303640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:17:17.825 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:17:17.824+0000 7f90e7303640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:17:17.833 INFO:teuthology.orchestra.run.smithi153.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-09-22T03:17:17.836 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-22T03:17:17.837 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-22T03:17:18.675 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-22T03:17:18.800 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:17:18.796+0000 7f92aca6c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:17:18.805 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:17:18.803+0000 7f92aca6c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:17:18.998 INFO:teuthology.orchestra.run.smithi153.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-09-22T03:17:19.001 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-22T03:17:19.001 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-22T03:17:20.621 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:Removed 2258 objects 2024-09-22T03:17:20.621 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:Clean up completed and total clean up time :6.61067 2024-09-22T03:17:21.002 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-22T03:17:21.108 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:17:21.105+0000 7f54f9797640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:17:21.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:17:21.109+0000 7f54f9797640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:17:21.165 INFO:teuthology.orchestra.run.smithi153.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-09-22T03:17:21.168 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-22T03:17:21.168 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-22T03:17:22.838 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [3] live_osds: [1, 0, 2] 2024-09-22T03:17:22.838 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-22T03:17:22.838 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-09-22T03:17:22.838 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-09-22T03:17:22.953 INFO:tasks.ceph.ceph_manager.ceph:removing pool_name unique_pool_4 2024-09-22T03:17:22.954 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm unique_pool_4 unique_pool_4 --yes-i-really-really-mean-it 2024-09-22T03:17:22.958 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:17:22.952+0000 7f7c9b2fa640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:17:22.961 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:17:22.960+0000 7f7c9b2fa640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:17:22.974 INFO:teuthology.orchestra.run.smithi153.stderr:no valid command found; 10 closest matches: 2024-09-22T03:17:22.974 INFO:teuthology.orchestra.run.smithi153.stderr:config get 2024-09-22T03:17:22.974 INFO:teuthology.orchestra.run.smithi153.stderr:config help 2024-09-22T03:17:22.974 INFO:teuthology.orchestra.run.smithi153.stderr:config set ... 2024-09-22T03:17:22.974 INFO:teuthology.orchestra.run.smithi153.stderr:config show 2024-09-22T03:17:22.974 INFO:teuthology.orchestra.run.smithi153.stderr:deep_scrub 2024-09-22T03:17:22.974 INFO:teuthology.orchestra.run.smithi153.stderr:dump_historic_ops 2024-09-22T03:17:22.975 INFO:teuthology.orchestra.run.smithi153.stderr:dump_historic_slow_ops 2024-09-22T03:17:22.975 INFO:teuthology.orchestra.run.smithi153.stderr:dump_metrics [] 2024-09-22T03:17:22.975 INFO:teuthology.orchestra.run.smithi153.stderr:dump_ops_in_flight 2024-09-22T03:17:22.975 INFO:teuthology.orchestra.run.smithi153.stderr:dump_pgstate_history 2024-09-22T03:17:22.975 INFO:teuthology.orchestra.run.smithi153.stderr:Error EINVAL: invalid command 2024-09-22T03:17:22.977 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-22T03:17:22.977 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-22T03:17:22.978 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-09-22T03:17:23.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:17:23.057+0000 7f9fc4997640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:17:23.070 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:17:23.068+0000 7f9fc4997640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:17:23.129 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:17:23.123+0000 7fa1d8ab2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:17:23.132 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:17:23.131+0000 7fa1d8ab2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:17:23.145 INFO:teuthology.orchestra.run.smithi153.stderr:no valid command found; 10 closest matches: 2024-09-22T03:17:23.145 INFO:teuthology.orchestra.run.smithi153.stderr:config get 2024-09-22T03:17:23.145 INFO:teuthology.orchestra.run.smithi153.stderr:config help 2024-09-22T03:17:23.145 INFO:teuthology.orchestra.run.smithi153.stderr:config set ... 2024-09-22T03:17:23.145 INFO:teuthology.orchestra.run.smithi153.stderr:config show 2024-09-22T03:17:23.146 INFO:teuthology.orchestra.run.smithi153.stderr:deep_scrub 2024-09-22T03:17:23.146 INFO:teuthology.orchestra.run.smithi153.stderr:dump_historic_ops 2024-09-22T03:17:23.146 INFO:teuthology.orchestra.run.smithi153.stderr:dump_historic_slow_ops 2024-09-22T03:17:23.146 INFO:teuthology.orchestra.run.smithi153.stderr:dump_metrics [] 2024-09-22T03:17:23.146 INFO:teuthology.orchestra.run.smithi153.stderr:dump_ops_in_flight 2024-09-22T03:17:23.146 INFO:teuthology.orchestra.run.smithi153.stderr:dump_pgstate_history 2024-09-22T03:17:23.146 INFO:teuthology.orchestra.run.smithi153.stderr:Error EINVAL: invalid command 2024-09-22T03:17:23.148 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-22T03:17:23.149 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-22T03:17:23.149 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-09-22T03:17:23.169 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-22T03:17:23.256 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:17:23.250+0000 7f39d644f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:17:23.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:17:23.258+0000 7f39d644f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:17:23.274 INFO:teuthology.orchestra.run.smithi153.stderr:no valid command found; 10 closest matches: 2024-09-22T03:17:23.274 INFO:teuthology.orchestra.run.smithi153.stderr:config get 2024-09-22T03:17:23.274 INFO:teuthology.orchestra.run.smithi153.stderr:config help 2024-09-22T03:17:23.274 INFO:teuthology.orchestra.run.smithi153.stderr:config set ... 2024-09-22T03:17:23.274 INFO:teuthology.orchestra.run.smithi153.stderr:config show 2024-09-22T03:17:23.274 INFO:teuthology.orchestra.run.smithi153.stderr:deep_scrub 2024-09-22T03:17:23.274 INFO:teuthology.orchestra.run.smithi153.stderr:dump_historic_ops 2024-09-22T03:17:23.274 INFO:teuthology.orchestra.run.smithi153.stderr:dump_historic_slow_ops 2024-09-22T03:17:23.274 INFO:teuthology.orchestra.run.smithi153.stderr:dump_metrics [] 2024-09-22T03:17:23.274 INFO:teuthology.orchestra.run.smithi153.stderr:dump_ops_in_flight 2024-09-22T03:17:23.274 INFO:teuthology.orchestra.run.smithi153.stderr:dump_pgstate_history 2024-09-22T03:17:23.274 INFO:teuthology.orchestra.run.smithi153.stderr:Error EINVAL: invalid command 2024-09-22T03:17:23.277 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-22T03:17:23.277 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-22T03:17:23.277 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-09-22T03:17:23.297 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:17:23.294+0000 7fa546b97640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:17:23.301 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:17:23.299+0000 7fa546b97640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:17:23.347 INFO:teuthology.orchestra.run.smithi153.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-09-22T03:17:23.351 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-22T03:17:23.351 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-22T03:17:23.386 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:17:23.380+0000 7fec7a397640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:17:23.389 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:17:23.388+0000 7fec7a397640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:17:23.393 INFO:teuthology.orchestra.run.smithi153.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-09-22T03:17:23.396 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-22T03:17:23.396 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-22T03:17:24.018 INFO:teuthology.orchestra.run.smithi153.stderr:pool 'unique_pool_4' removed 2024-09-22T03:17:24.036 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-09-22T03:17:24.037 INFO:tasks.radosbench:Beginning radosbench... 2024-09-22T03:17:24.037 INFO:tasks.ceph.ceph_manager.ceph:creating pool_name unique_pool_5 2024-09-22T03:17:24.037 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create unique_pool_5 16 2024-09-22T03:17:24.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:17:24.167+0000 7f8bf0f17640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:17:24.182 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:17:24.180+0000 7f8bf0f17640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:17:25.024 INFO:teuthology.orchestra.run.smithi153.stderr:pool 'unique_pool_5' created 2024-09-22T03:17:25.043 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool application enable unique_pool_5 rados --yes-i-really-mean-it || true 2024-09-22T03:17:25.165 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:17:25.155+0000 7f6255397640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:17:25.168 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:17:25.167+0000 7f6255397640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:17:25.352 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-22T03:17:25.454 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:17:25.451+0000 7ff87946c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:17:25.457 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:17:25.456+0000 7ff87946c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:17:25.502 INFO:teuthology.orchestra.run.smithi153.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-09-22T03:17:25.504 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-22T03:17:25.505 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-22T03:17:26.025 INFO:teuthology.orchestra.run.smithi153.stderr:enabled application 'rados' on pool 'unique_pool_5' 2024-09-22T03:17:27.044 DEBUG:teuthology.orchestra.run.smithi153:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 8192 --object-size 65536 -p unique_pool_5 bench 90 write' 2024-09-22T03:17:27.047 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-09-22T03:17:27.047 INFO:teuthology.orchestra.run:waiting for 3000 2024-09-22T03:17:27.105 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:hints = 1 2024-09-22T03:17:27.105 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:Maintaining 16 concurrent writes of 8192 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-09-22T03:17:27.105 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:Object prefix: benchmark_data_smithi153_41696 2024-09-22T03:17:27.105 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-22T03:17:27.105 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 0 0 0 0 0 0 - 0 2024-09-22T03:17:27.506 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-22T03:17:27.639 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:17:27.635+0000 7f177e26c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:17:27.644 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:17:27.642+0000 7f177e26c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:17:27.723 INFO:teuthology.orchestra.run.smithi153.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-09-22T03:17:27.727 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-22T03:17:27.728 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-22T03:17:28.105 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 1 16 363 347 2.71078 2.71094 0.046153 0.0449151 2024-09-22T03:17:28.397 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [3] live_osds: [1, 0, 2] 2024-09-22T03:17:28.397 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-22T03:17:28.397 INFO:tasks.thrashosds.thrasher:Killing osd 1, live_osds are [1, 0, 2] 2024-09-22T03:17:28.397 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-09-22T03:17:28.397 INFO:teuthology.orchestra.run:waiting for 300 2024-09-22T03:17:29.105 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 2 16 469 453 1.76937 0.828125 0.0158611 0.0455442 2024-09-22T03:17:29.729 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-22T03:17:29.844 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:17:29.841+0000 7fa8de46c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:17:29.847 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:17:29.846+0000 7fa8de46c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:17:30.105 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 3 16 469 453 1.17955 0 - 0.0455442 2024-09-22T03:17:31.105 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 4 16 469 453 0.884657 0 - 0.0455442 2024-09-22T03:17:31.937 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~0s 2024-09-22T03:17:32.105 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 5 16 469 453 0.707726 0 - 0.0455442 2024-09-22T03:17:33.105 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 6 16 469 453 0.589771 0 - 0.0455442 2024-09-22T03:17:34.106 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 7 16 469 453 0.505521 0 - 0.0455442 2024-09-22T03:17:34.498 INFO:tasks.ceph.osd.1:Stopped 2024-09-22T03:17:35.106 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 8 16 469 453 0.442327 0 - 0.0455442 2024-09-22T03:17:36.106 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 9 16 469 453 0.393181 0 - 0.0455442 2024-09-22T03:17:37.106 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 10 16 469 453 0.353862 0 - 0.0455442 2024-09-22T03:17:37.341 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 has been restored 2024-09-22T03:17:38.106 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 11 16 469 453 0.321692 0 - 0.0455442 2024-09-22T03:17:39.107 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 12 16 469 453 0.294883 0 - 0.0455442 2024-09-22T03:17:39.498 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [3, 1] live_osds: [0, 2] 2024-09-22T03:17:39.498 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-22T03:17:39.499 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-22T03:17:39.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:17:39.611+0000 7fdec82b2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:17:39.620 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:17:39.619+0000 7fdec82b2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:17:39.797 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-22T03:17:39.797 INFO:teuthology.orchestra.run.smithi153.stderr:dumped all 2024-09-22T03:17:39.809 INFO:teuthology.orchestra.run.smithi153.stdout:{"pg_ready":true,"pg_map":{"version":397,"stamp":"2024-09-22T03:17:38.229658+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3580467,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":504,"num_write_kb":4816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":453,"ondisk_log_size":453,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":268468,"kb_used_data":63284,"kb_used_omap":0,"kb_used_meta":205120,"kb_avail":377218892,"statfs":{"total":386547056640,"available":386272145408,"internally_reserved":0,"allocated":64802816,"data_stored":144517913,"data_compressed":7293817,"data_compressed_allocated":60907520,"data_compressed_original":142942208,"omap_allocated":0,"internal_metadata":210042880},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2990080,"num_objects":46,"num_object_clones":0,"num_object_copies":138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":365,"num_write_kb":2920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.195915"},"pg_stats":[{"pgid":"8.7","version":"116'47","reported_seq":57,"reported_epoch":116,"state":"active+undersized+degraded","last_fresh":"2024-09-22T03:17:28.198203+0000","last_change":"2024-09-22T03:17:27.418734+0000","last_active":"2024-09-22T03:17:28.198203+0000","last_peered":"2024-09-22T03:17:28.198203+0000","last_clean":"2024-09-22T03:17:25.017919+0000","last_became_active":"2024-09-22T03:17:26.072489+0000","last_became_peered":"2024-09-22T03:17:26.072489+0000","last_unstale":"2024-09-22T03:17:28.198203+0000","last_undegraded":"2024-09-22T03:17:27.369885+0000","last_fullsized":"2024-09-22T03:17:26.030871+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:17:25.017919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:17:25.017919+0000","last_clean_scrub_stamp":"2024-09-22T03:17:25.017919+0000","objects_scrubbed":0,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48,"num_write_kb":384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":6}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.6","version":"116'39","reported_seq":52,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-09-22T03:17:28.356828+0000","last_change":"2024-09-22T03:17:26.094227+0000","last_active":"2024-09-22T03:17:28.356828+0000","last_peered":"2024-09-22T03:17:28.356828+0000","last_clean":"2024-09-22T03:17:28.356828+0000","last_became_active":"2024-09-22T03:17:26.091228+0000","last_became_peered":"2024-09-22T03:17:26.091228+0000","last_unstale":"2024-09-22T03:17:28.356828+0000","last_undegraded":"2024-09-22T03:17:28.356828+0000","last_fullsized":"2024-09-22T03:17:28.356828+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:17:25.017919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:17:25.017919+0000","last_clean_scrub_stamp":"2024-09-22T03:17:25.017919+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":40,"num_write_kb":320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-22T03:17:25.017919+0000","last_change":"2024-09-22T03:17:25.017919+0000","last_active":"2024-09-22T03:17:25.017919+0000","last_peered":"2024-09-22T03:17:25.017919+0000","last_clean":"2024-09-22T03:17:25.017919+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-22T03:17:25.017919+0000","last_undegraded":"2024-09-22T03:17:25.017919+0000","last_fullsized":"2024-09-22T03:17:25.017919+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:17:25.017919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:17:25.017919+0000","last_clean_scrub_stamp":"2024-09-22T03:17:25.017919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.4","version":"116'23","reported_seq":33,"reported_epoch":116,"state":"active+undersized+degraded","last_fresh":"2024-09-22T03:17:28.114471+0000","last_change":"2024-09-22T03:17:27.149165+0000","last_active":"2024-09-22T03:17:28.114471+0000","last_peered":"2024-09-22T03:17:28.114471+0000","last_clean":"2024-09-22T03:17:25.017919+0000","last_became_active":"2024-09-22T03:17:26.101059+0000","last_became_peered":"2024-09-22T03:17:26.101059+0000","last_unstale":"2024-09-22T03:17:28.114471+0000","last_undegraded":"2024-09-22T03:17:27.146196+0000","last_fullsized":"2024-09-22T03:17:26.049819+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:17:25.017919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:17:25.017919+0000","last_clean_scrub_stamp":"2024-09-22T03:17:25.017919+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196608,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24,"num_write_kb":192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":3}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-22T03:17:25.017919+0000","last_change":"2024-09-22T03:17:25.017919+0000","last_active":"2024-09-22T03:17:25.017919+0000","last_peered":"2024-09-22T03:17:25.017919+0000","last_clean":"2024-09-22T03:17:25.017919+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-22T03:17:25.017919+0000","last_undegraded":"2024-09-22T03:17:25.017919+0000","last_fullsized":"2024-09-22T03:17:25.017919+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:17:25.017919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:17:25.017919+0000","last_clean_scrub_stamp":"2024-09-22T03:17:25.017919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.2","version":"116'47","reported_seq":57,"reported_epoch":116,"state":"active+undersized+degraded","last_fresh":"2024-09-22T03:17:28.423620+0000","last_change":"2024-09-22T03:17:27.174610+0000","last_active":"2024-09-22T03:17:28.423620+0000","last_peered":"2024-09-22T03:17:28.423620+0000","last_clean":"2024-09-22T03:17:25.017919+0000","last_became_active":"2024-09-22T03:17:26.076847+0000","last_became_peered":"2024-09-22T03:17:26.076847+0000","last_unstale":"2024-09-22T03:17:28.423620+0000","last_undegraded":"2024-09-22T03:17:27.167770+0000","last_fullsized":"2024-09-22T03:17:26.039238+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:17:25.017919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:17:25.017919+0000","last_clean_scrub_stamp":"2024-09-22T03:17:25.017919+0000","objects_scrubbed":0,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48,"num_write_kb":384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,2","objects":6}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1","version":"116'39","reported_seq":49,"reported_epoch":116,"state":"active+undersized+degraded","last_fresh":"2024-09-22T03:17:27.926858+0000","last_change":"2024-09-22T03:17:27.411274+0000","last_active":"2024-09-22T03:17:27.926858+0000","last_peered":"2024-09-22T03:17:27.926858+0000","last_clean":"2024-09-22T03:17:25.017919+0000","last_became_active":"2024-09-22T03:17:26.096823+0000","last_became_peered":"2024-09-22T03:17:26.096823+0000","last_unstale":"2024-09-22T03:17:27.926858+0000","last_undegraded":"2024-09-22T03:17:27.405554+0000","last_fullsized":"2024-09-22T03:17:26.048596+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:17:25.017919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:17:25.017919+0000","last_clean_scrub_stamp":"2024-09-22T03:17:25.017919+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":40,"num_write_kb":320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":5}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-22T03:17:25.017919+0000","last_change":"2024-09-22T03:17:25.017919+0000","last_active":"2024-09-22T03:17:25.017919+0000","last_peered":"2024-09-22T03:17:25.017919+0000","last_clean":"2024-09-22T03:17:25.017919+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-22T03:17:25.017919+0000","last_undegraded":"2024-09-22T03:17:25.017919+0000","last_fullsized":"2024-09-22T03:17:25.017919+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:17:25.017919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:17:25.017919+0000","last_clean_scrub_stamp":"2024-09-22T03:17:25.017919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.9","version":"116'47","reported_seq":57,"reported_epoch":116,"state":"active+undersized+degraded","last_fresh":"2024-09-22T03:17:27.967124+0000","last_change":"2024-09-22T03:17:27.213454+0000","last_active":"2024-09-22T03:17:27.967124+0000","last_peered":"2024-09-22T03:17:27.967124+0000","last_clean":"2024-09-22T03:17:25.017919+0000","last_became_active":"2024-09-22T03:17:26.070849+0000","last_became_peered":"2024-09-22T03:17:26.070849+0000","last_unstale":"2024-09-22T03:17:27.967124+0000","last_undegraded":"2024-09-22T03:17:27.209797+0000","last_fullsized":"2024-09-22T03:17:26.041980+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:17:25.017919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:17:25.017919+0000","last_clean_scrub_stamp":"2024-09-22T03:17:25.017919+0000","objects_scrubbed":0,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48,"num_write_kb":384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,2","objects":6}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":148,"reported_epoch":116,"state":"active+undersized","last_fresh":"2024-09-22T03:17:27.044847+0000","last_change":"2024-09-22T03:17:12.997162+0000","last_active":"2024-09-22T03:17:27.044847+0000","last_peered":"2024-09-22T03:17:27.044847+0000","last_clean":"2024-09-22T03:15:51.004319+0000","last_became_active":"2024-09-22T03:17:12.997162+0000","last_became_peered":"2024-09-22T03:17:12.997162+0000","last_unstale":"2024-09-22T03:17:27.044847+0000","last_undegraded":"2024-09-22T03:17:27.044847+0000","last_fullsized":"2024-09-22T03:17:12.958168+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-22T03:17:25.017919+0000","last_change":"2024-09-22T03:17:25.017919+0000","last_active":"2024-09-22T03:17:25.017919+0000","last_peered":"2024-09-22T03:17:25.017919+0000","last_clean":"2024-09-22T03:17:25.017919+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-22T03:17:25.017919+0000","last_undegraded":"2024-09-22T03:17:25.017919+0000","last_fullsized":"2024-09-22T03:17:25.017919+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:17:25.017919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:17:25.017919+0000","last_clean_scrub_stamp":"2024-09-22T03:17:25.017919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":269,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-09-22T03:17:27.042392+0000","last_change":"2024-09-22T03:16:18.925677+0000","last_active":"2024-09-22T03:17:27.042392+0000","last_peered":"2024-09-22T03:17:27.042392+0000","last_clean":"2024-09-22T03:17:27.042392+0000","last_became_active":"2024-09-22T03:16:17.986660+0000","last_became_peered":"2024-09-22T03:16:17.986660+0000","last_unstale":"2024-09-22T03:17:27.042392+0000","last_undegraded":"2024-09-22T03:17:27.042392+0000","last_fullsized":"2024-09-22T03:17:27.042392+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.b","version":"116'36","reported_seq":49,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-09-22T03:17:28.440073+0000","last_change":"2024-09-22T03:17:26.134397+0000","last_active":"2024-09-22T03:17:28.440073+0000","last_peered":"2024-09-22T03:17:28.440073+0000","last_clean":"2024-09-22T03:17:28.440073+0000","last_became_active":"2024-09-22T03:17:26.131140+0000","last_became_peered":"2024-09-22T03:17:26.131140+0000","last_unstale":"2024-09-22T03:17:28.440073+0000","last_undegraded":"2024-09-22T03:17:28.440073+0000","last_fullsized":"2024-09-22T03:17:28.440073+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:17:25.017919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:17:25.017919+0000","last_clean_scrub_stamp":"2024-09-22T03:17:25.017919+0000","objects_scrubbed":0,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":303104,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":37,"num_write_kb":296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":263,"reported_epoch":116,"state":"active+undersized+degraded","last_fresh":"2024-09-22T03:17:27.057454+0000","last_change":"2024-09-22T03:17:13.008275+0000","last_active":"2024-09-22T03:17:27.057454+0000","last_peered":"2024-09-22T03:17:27.057454+0000","last_clean":"2024-09-22T03:16:38.920818+0000","last_became_active":"2024-09-22T03:17:13.008275+0000","last_became_peered":"2024-09-22T03:17:13.008275+0000","last_unstale":"2024-09-22T03:17:27.057454+0000","last_undegraded":"2024-09-22T03:17:12.963487+0000","last_fullsized":"2024-09-22T03:17:12.962662+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.a","version":"116'15","reported_seq":25,"reported_epoch":116,"state":"active+undersized+degraded","last_fresh":"2024-09-22T03:17:28.396031+0000","last_change":"2024-09-22T03:17:27.329705+0000","last_active":"2024-09-22T03:17:28.396031+0000","last_peered":"2024-09-22T03:17:28.396031+0000","last_clean":"2024-09-22T03:17:25.017919+0000","last_became_active":"2024-09-22T03:17:26.076237+0000","last_became_peered":"2024-09-22T03:17:26.076237+0000","last_unstale":"2024-09-22T03:17:28.396031+0000","last_undegraded":"2024-09-22T03:17:27.324442+0000","last_fullsized":"2024-09-22T03:17:26.032593+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:17:25.017919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:17:25.017919+0000","last_clean_scrub_stamp":"2024-09-22T03:17:25.017919+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131072,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":2}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":50,"reported_epoch":113,"state":"active+undersized+degraded","last_fresh":"2024-09-22T03:17:24.036724+0000","last_change":"2024-09-22T03:17:13.032574+0000","last_active":"2024-09-22T03:17:24.036724+0000","last_peered":"2024-09-22T03:17:24.036724+0000","last_clean":"2024-09-22T03:08:04.133992+0000","last_became_active":"2024-09-22T03:17:13.032574+0000","last_became_peered":"2024-09-22T03:17:13.032574+0000","last_unstale":"2024-09-22T03:17:24.036724+0000","last_undegraded":"2024-09-22T03:17:12.963807+0000","last_fullsized":"2024-09-22T03:17:12.962634+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"12'97","reported_seq":391,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-09-22T03:17:27.045238+0000","last_change":"2024-09-22T03:16:17.998607+0000","last_active":"2024-09-22T03:17:27.045238+0000","last_peered":"2024-09-22T03:17:27.045238+0000","last_clean":"2024-09-22T03:17:27.045238+0000","last_became_active":"2024-09-22T03:16:17.995046+0000","last_became_peered":"2024-09-22T03:16:17.995046+0000","last_unstale":"2024-09-22T03:17:27.045238+0000","last_undegraded":"2024-09-22T03:17:27.045238+0000","last_fullsized":"2024-09-22T03:17:27.045238+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.d","version":"116'23","reported_seq":36,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-09-22T03:17:28.157449+0000","last_change":"2024-09-22T03:17:26.129326+0000","last_active":"2024-09-22T03:17:28.157449+0000","last_peered":"2024-09-22T03:17:28.157449+0000","last_clean":"2024-09-22T03:17:28.157449+0000","last_became_active":"2024-09-22T03:17:26.126415+0000","last_became_peered":"2024-09-22T03:17:26.126415+0000","last_unstale":"2024-09-22T03:17:28.157449+0000","last_undegraded":"2024-09-22T03:17:28.157449+0000","last_fullsized":"2024-09-22T03:17:28.157449+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:17:25.017919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:17:25.017919+0000","last_clean_scrub_stamp":"2024-09-22T03:17:25.017919+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196608,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24,"num_write_kb":192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":265,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-22T03:17:24.060110+0000","last_change":"2024-09-22T03:16:18.925553+0000","last_active":"2024-09-22T03:17:24.060110+0000","last_peered":"2024-09-22T03:17:24.060110+0000","last_clean":"2024-09-22T03:17:24.060110+0000","last_became_active":"2024-09-22T03:16:18.021400+0000","last_became_peered":"2024-09-22T03:16:18.021400+0000","last_unstale":"2024-09-22T03:17:24.060110+0000","last_undegraded":"2024-09-22T03:17:24.060110+0000","last_fullsized":"2024-09-22T03:17:24.060110+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.c","version":"116'15","reported_seq":25,"reported_epoch":116,"state":"active+undersized+degraded","last_fresh":"2024-09-22T03:17:27.780618+0000","last_change":"2024-09-22T03:17:27.508654+0000","last_active":"2024-09-22T03:17:27.780618+0000","last_peered":"2024-09-22T03:17:27.780618+0000","last_clean":"2024-09-22T03:17:25.017919+0000","last_became_active":"2024-09-22T03:17:26.082313+0000","last_became_peered":"2024-09-22T03:17:26.082313+0000","last_unstale":"2024-09-22T03:17:27.780618+0000","last_undegraded":"2024-09-22T03:17:27.505126+0000","last_fullsized":"2024-09-22T03:17:26.044948+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:17:25.017919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:17:25.017919+0000","last_clean_scrub_stamp":"2024-09-22T03:17:25.017919+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131072,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":2}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":242,"reported_epoch":116,"state":"active+undersized","last_fresh":"2024-09-22T03:17:27.058027+0000","last_change":"2024-09-22T03:17:12.989359+0000","last_active":"2024-09-22T03:17:27.058027+0000","last_peered":"2024-09-22T03:17:27.058027+0000","last_clean":"2024-09-22T03:16:38.920894+0000","last_became_active":"2024-09-22T03:17:12.989359+0000","last_became_peered":"2024-09-22T03:17:12.989359+0000","last_unstale":"2024-09-22T03:17:27.058027+0000","last_undegraded":"2024-09-22T03:17:27.058027+0000","last_fullsized":"2024-09-22T03:17:12.964257+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":["2","0"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.f","version":"116'7","reported_seq":17,"reported_epoch":116,"state":"active+undersized+degraded","last_fresh":"2024-09-22T03:17:27.134555+0000","last_change":"2024-09-22T03:17:27.124632+0000","last_active":"2024-09-22T03:17:27.134555+0000","last_peered":"2024-09-22T03:17:27.134555+0000","last_clean":"2024-09-22T03:17:25.017919+0000","last_became_active":"2024-09-22T03:17:26.086629+0000","last_became_peered":"2024-09-22T03:17:26.086629+0000","last_unstale":"2024-09-22T03:17:27.134555+0000","last_undegraded":"2024-09-22T03:17:27.114315+0000","last_fullsized":"2024-09-22T03:17:26.037534+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:17:25.017919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:17:25.017919+0000","last_clean_scrub_stamp":"2024-09-22T03:17:25.017919+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65536,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":279,"reported_epoch":113,"state":"active+undersized","last_fresh":"2024-09-22T03:17:24.054350+0000","last_change":"2024-09-22T03:17:13.011522+0000","last_active":"2024-09-22T03:17:24.054350+0000","last_peered":"2024-09-22T03:17:24.054350+0000","last_clean":"2024-09-22T03:16:38.929075+0000","last_became_active":"2024-09-22T03:17:13.011522+0000","last_became_peered":"2024-09-22T03:17:13.011522+0000","last_unstale":"2024-09-22T03:17:24.054350+0000","last_undegraded":"2024-09-22T03:17:24.054350+0000","last_fullsized":"2024-09-22T03:17:12.965814+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.e","version":"116'15","reported_seq":25,"reported_epoch":116,"state":"active+undersized+degraded","last_fresh":"2024-09-22T03:17:28.284763+0000","last_change":"2024-09-22T03:17:27.116911+0000","last_active":"2024-09-22T03:17:28.284763+0000","last_peered":"2024-09-22T03:17:28.284763+0000","last_clean":"2024-09-22T03:17:25.017919+0000","last_became_active":"2024-09-22T03:17:26.107675+0000","last_became_peered":"2024-09-22T03:17:26.107675+0000","last_unstale":"2024-09-22T03:17:28.284763+0000","last_undegraded":"2024-09-22T03:17:27.111955+0000","last_fullsized":"2024-09-22T03:17:26.053392+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:17:25.017919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:17:25.017919+0000","last_clean_scrub_stamp":"2024-09-22T03:17:25.017919+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131072,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":2}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":272,"reported_epoch":113,"state":"active+undersized","last_fresh":"2024-09-22T03:17:24.052232+0000","last_change":"2024-09-22T03:17:13.046376+0000","last_active":"2024-09-22T03:17:24.052232+0000","last_peered":"2024-09-22T03:17:24.052232+0000","last_clean":"2024-09-22T03:16:38.928974+0000","last_became_active":"2024-09-22T03:17:13.046376+0000","last_became_peered":"2024-09-22T03:17:13.046376+0000","last_unstale":"2024-09-22T03:17:24.052232+0000","last_undegraded":"2024-09-22T03:17:24.052232+0000","last_fullsized":"2024-09-22T03:17:12.962204+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":2990080,"num_objects":46,"num_object_clones":0,"num_object_copies":138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":365,"num_write_kb":2920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2859008,"data_stored":5718016,"data_compressed":294390,"data_compressed_allocated":2859008,"data_compressed_original":5718016,"omap_allocated":0,"internal_metadata":0},"log_size":353,"ondisk_log_size":353,"up":27,"acting":27,"num_store_stats":2},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1180736,"data_compressed":59924,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":18,"acting":18,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":87,"seq":373662154827,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":117136,"kb_used_data":57472,"kb_used_omap":0,"kb_used_meta":59648,"kb_avail":94254704,"statfs":{"total":96636764160,"available":96516816896,"internally_reserved":0,"allocated":58851328,"data_stored":135871990,"data_compressed":6909541,"data_compressed_allocated":57925632,"data_compressed_original":135503872,"omap_allocated":0,"internal_metadata":61079552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":100,"seq":429496729679,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":47292,"kb_used_data":2220,"kb_used_omap":0,"kb_used_meta":45056,"kb_avail":94324548,"statfs":{"total":96636764160,"available":96588337152,"internally_reserved":0,"allocated":2273280,"data_stored":3445380,"data_compressed":156900,"data_compressed_allocated":1273856,"data_compressed_original":3039232,"omap_allocated":0,"internal_metadata":46137344},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":53,"seq":227633266977,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":51248,"kb_used_data":992,"kb_used_omap":0,"kb_used_meta":50240,"kb_avail":94320592,"statfs":{"total":96636764160,"available":96584286208,"internally_reserved":0,"allocated":1015808,"data_stored":968750,"data_compressed":29962,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":51445760},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":73,"seq":313532612788,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":52792,"kb_used_data":2600,"kb_used_omap":0,"kb_used_meta":50176,"kb_avail":94319048,"statfs":{"total":96636764160,"available":96582705152,"internally_reserved":0,"allocated":2662400,"data_stored":4231793,"data_compressed":197414,"data_compressed_allocated":1667072,"data_compressed_original":3825664,"omap_allocated":0,"internal_metadata":51380224},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29962,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29962,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1626112,"data_stored":3252224,"data_compressed":167452,"data_compressed_allocated":1626112,"data_compressed_original":3252224,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1232896,"data_stored":2465792,"data_compressed":126938,"data_compressed_allocated":1232896,"data_compressed_original":2465792,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-22T03:17:39.811 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg force-recovery 8.7 8.2 8.1 8.9 1.2 8.f 8.e 2024-09-22T03:17:39.922 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:17:39.909+0000 7fd2f1357640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:17:39.925 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:17:39.923+0000 7fd2f1357640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:17:40.098 INFO:teuthology.orchestra.run.smithi153.stderr:instructing pg(s) [8.1,8.2,8.9,8.e] on osd.0 to force-recovery; instructing pg(s) [1.2,8.7,8.f] on osd.2 to force-recovery; 2024-09-22T03:17:40.098 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-22T03:17:40.106 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 13 16 469 453 0.272201 0 - 0.0455442 2024-09-22T03:17:41.107 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 14 16 469 453 0.252757 0 - 0.0455442 2024-09-22T03:17:42.107 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 15 16 469 453 0.235908 0 - 0.0455442 2024-09-22T03:17:43.107 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 16 16 469 453 0.221164 0 - 0.0455442 2024-09-22T03:17:44.107 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 17 16 469 453 0.208154 0 - 0.0455442 2024-09-22T03:17:45.107 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 18 16 469 453 0.196589 0 - 0.0455442 2024-09-22T03:17:45.112 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [3, 1] live_osds: [0, 2] 2024-09-22T03:17:45.112 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-22T03:17:45.112 INFO:tasks.thrashosds.thrasher:Reviving osd 3 2024-09-22T03:17:45.112 INFO:tasks.ceph.osd.3:Restarting daemon 2024-09-22T03:17:45.112 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2>> /var/log/ceph/ceph-osd.3.log 2024-09-22T03:17:45.115 INFO:tasks.ceph.osd.3:Started 2024-09-22T03:17:45.115 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-22T03:17:45.225 INFO:teuthology.orchestra.run.smithi196.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-09-22T03:17:45.228 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-22T03:17:45.229 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-09-22T03:17:46.107 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 19 16 469 453 0.186242 0 - 0.0455442 2024-09-22T03:17:47.108 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:2024-09-22T03:17:47.107071+0000 min lat: 0.0113502 max lat: 0.101899 avg lat: 0.0455442 2024-09-22T03:17:47.108 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-22T03:17:47.108 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 20 16 469 453 0.17693 0 - 0.0455442 2024-09-22T03:17:48.108 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 21 16 469 453 0.168504 0 - 0.0455442 2024-09-22T03:17:49.108 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 22 16 469 453 0.160845 0 - 0.0455442 2024-09-22T03:17:50.108 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 23 16 469 453 0.153852 0 - 0.0455442 2024-09-22T03:17:50.230 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-22T03:17:50.325 INFO:teuthology.orchestra.run.smithi196.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-09-22T03:17:50.328 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-22T03:17:50.328 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-09-22T03:17:51.108 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 24 16 469 453 0.147441 0 - 0.0455442 2024-09-22T03:17:52.108 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 25 16 469 453 0.141543 0 - 0.0455442 2024-09-22T03:17:53.108 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 26 16 469 453 0.136099 0 - 0.0455442 2024-09-22T03:17:54.109 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 27 16 469 453 0.131058 0 - 0.0455442 2024-09-22T03:17:55.109 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 28 16 469 453 0.126378 0 - 0.0455442 2024-09-22T03:17:55.329 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-22T03:17:55.425 INFO:teuthology.orchestra.run.smithi196.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-09-22T03:17:55.428 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-22T03:17:55.428 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-09-22T03:17:56.109 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 29 16 469 453 0.12202 0 - 0.0455442 2024-09-22T03:17:57.109 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 30 16 469 453 0.117953 0 - 0.0455442 2024-09-22T03:17:57.934 INFO:teuthology.orchestra.run.smithi153.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-09-22T03:17:57.999 INFO:teuthology.orchestra.run.smithi153.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-09-22T03:17:58.003 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-22T03:17:58.004 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-22T03:17:58.109 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 31 16 469 453 0.114148 0 - 0.0455442 2024-09-22T03:17:59.109 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 32 16 479 463 0.113022 0.00260417 0.0319037 0.506901 2024-09-22T03:18:00.005 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-22T03:18:00.109 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 33 16 479 463 0.109597 0 - 0.506901 2024-09-22T03:18:00.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:18:00.108+0000 7fabf5b97640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:18:00.114 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:18:00.112+0000 7fabf5b97640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:18:00.131 INFO:teuthology.orchestra.run.smithi153.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-09-22T03:18:00.186 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-22T03:18:00.187 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-22T03:18:00.429 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-22T03:18:00.551 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-09-22T03:18:00.551 INFO:teuthology.orchestra.run.smithi196.stdout: "ops_in_flight": [ 2024-09-22T03:18:00.551 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-22T03:18:00.552 INFO:teuthology.orchestra.run.smithi196.stdout: "type": "peering_event", 2024-09-22T03:18:00.552 INFO:teuthology.orchestra.run.smithi196.stdout: "id": 78, 2024-09-22T03:18:00.552 INFO:teuthology.orchestra.run.smithi196.stdout: "detail": { 2024-09-22T03:18:00.552 INFO:teuthology.orchestra.run.smithi196.stdout: "PeeringEvent": { 2024-09-22T03:18:00.552 INFO:teuthology.orchestra.run.smithi196.stdout: "from": "3", 2024-09-22T03:18:00.552 INFO:teuthology.orchestra.run.smithi196.stdout: "pgid": "7.1", 2024-09-22T03:18:00.552 INFO:teuthology.orchestra.run.smithi196.stdout: "sent": 119, 2024-09-22T03:18:00.552 INFO:teuthology.orchestra.run.smithi196.stdout: "requested": 119, 2024-09-22T03:18:00.552 INFO:teuthology.orchestra.run.smithi196.stdout: "evt": "epoch_sent: 119 epoch_requested: 119 DeleteSome", 2024-09-22T03:18:00.552 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-22T03:18:00.552 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-22T03:18:00.552 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:18:00.552 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:00.553 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:00.553 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-22T03:18:00.553 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGPeeringPipeline::AwaitMap", 2024-09-22T03:18:00.553 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:18:00.540477+0000" 2024-09-22T03:18:00.553 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:00.553 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-22T03:18:00.553 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:18:00.553 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:00.553 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:00.553 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-22T03:18:00.553 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:18:00.553 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:00.553 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:00.553 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-22T03:18:00.554 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:18:00.554 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:00.554 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:00.554 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-22T03:18:00.554 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGPeeringPipeline::Process", 2024-09-22T03:18:00.554 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:18:00.540516+0000" 2024-09-22T03:18:00.554 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:00.554 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-22T03:18:00.554 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:18:00.554 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:00.554 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:00.554 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-22T03:18:00.554 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:18:00.555 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:00.555 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:18:00.555 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-22T03:18:00.555 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:18:00.555 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:18:00.555 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:00.555 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-22T03:18:00.555 INFO:teuthology.orchestra.run.smithi196.stdout: "type": "peering_event", 2024-09-22T03:18:00.555 INFO:teuthology.orchestra.run.smithi196.stdout: "id": 16777286, 2024-09-22T03:18:00.555 INFO:teuthology.orchestra.run.smithi196.stdout: "detail": { 2024-09-22T03:18:00.555 INFO:teuthology.orchestra.run.smithi196.stdout: "PeeringEvent": { 2024-09-22T03:18:00.555 INFO:teuthology.orchestra.run.smithi196.stdout: "from": "3", 2024-09-22T03:18:00.555 INFO:teuthology.orchestra.run.smithi196.stdout: "pgid": "7.c", 2024-09-22T03:18:00.555 INFO:teuthology.orchestra.run.smithi196.stdout: "sent": 119, 2024-09-22T03:18:00.556 INFO:teuthology.orchestra.run.smithi196.stdout: "requested": 119, 2024-09-22T03:18:00.556 INFO:teuthology.orchestra.run.smithi196.stdout: "evt": "epoch_sent: 119 epoch_requested: 119 DeleteSome", 2024-09-22T03:18:00.556 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-22T03:18:00.556 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-22T03:18:00.556 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:18:00.556 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:00.556 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:00.556 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-22T03:18:00.556 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGPeeringPipeline::AwaitMap", 2024-09-22T03:18:00.556 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:18:00.526695+0000" 2024-09-22T03:18:00.556 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:00.556 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-22T03:18:00.556 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:18:00.556 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:00.556 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:00.557 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-22T03:18:00.557 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:18:00.557 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:00.557 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:00.557 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-22T03:18:00.557 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:18:00.557 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:00.557 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:00.557 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-22T03:18:00.557 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGPeeringPipeline::Process", 2024-09-22T03:18:00.557 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:18:00.526734+0000" 2024-09-22T03:18:00.557 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:00.557 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-22T03:18:00.557 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:18:00.557 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:00.558 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:00.558 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-22T03:18:00.558 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:18:00.558 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:00.558 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:18:00.558 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-22T03:18:00.558 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:18:00.558 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:18:00.558 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:00.558 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-22T03:18:00.558 INFO:teuthology.orchestra.run.smithi196.stdout: "type": "peering_event", 2024-09-22T03:18:00.558 INFO:teuthology.orchestra.run.smithi196.stdout: "id": 33554479, 2024-09-22T03:18:00.558 INFO:teuthology.orchestra.run.smithi196.stdout: "detail": { 2024-09-22T03:18:00.558 INFO:teuthology.orchestra.run.smithi196.stdout: "PeeringEvent": { 2024-09-22T03:18:00.559 INFO:teuthology.orchestra.run.smithi196.stdout: "from": "3", 2024-09-22T03:18:00.559 INFO:teuthology.orchestra.run.smithi196.stdout: "pgid": "7.7", 2024-09-22T03:18:00.559 INFO:teuthology.orchestra.run.smithi196.stdout: "sent": 119, 2024-09-22T03:18:00.559 INFO:teuthology.orchestra.run.smithi196.stdout: "requested": 119, 2024-09-22T03:18:00.559 INFO:teuthology.orchestra.run.smithi196.stdout: "evt": "epoch_sent: 119 epoch_requested: 119 DeleteSome", 2024-09-22T03:18:00.559 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-09-22T03:18:00.559 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-22T03:18:00.559 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:18:00.559 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:00.559 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:00.559 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-22T03:18:00.559 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGPeeringPipeline::AwaitMap", 2024-09-22T03:18:00.559 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:18:00.416599+0000" 2024-09-22T03:18:00.559 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:00.559 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-22T03:18:00.560 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:18:00.560 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:00.560 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:00.560 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-22T03:18:00.560 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:18:00.560 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:00.560 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:00.560 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-22T03:18:00.560 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:18:00.560 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:00.560 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:00.560 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-22T03:18:00.560 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGPeeringPipeline::Process", 2024-09-22T03:18:00.560 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:18:00.416626+0000" 2024-09-22T03:18:00.561 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:00.561 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-22T03:18:00.561 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:18:00.561 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:00.561 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:00.561 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-22T03:18:00.561 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:18:00.561 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:00.561 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:18:00.561 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-22T03:18:00.561 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:18:00.561 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:18:00.561 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:18:00.561 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-22T03:18:00.561 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-09-22T03:18:01.110 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 34 16 479 463 0.106373 0 - 0.506901 2024-09-22T03:18:02.110 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 35 16 639 623 0.139044 0.416667 0.179484 0.898023 2024-09-22T03:18:02.187 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-22T03:18:02.305 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:18:02.302+0000 7f2fdcb97640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:18:02.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:18:02.308+0000 7f2fdcb97640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:18:02.397 INFO:teuthology.orchestra.run.smithi153.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-09-22T03:18:02.512 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-22T03:18:02.513 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-22T03:18:03.110 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 36 16 1009 993 0.215466 2.89062 0.0280307 0.579874 2024-09-22T03:18:04.110 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 37 16 1441 1425 0.300847 3.375 0.0163304 0.41531 2024-09-22T03:18:04.514 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-22T03:18:04.630 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:18:04.627+0000 7fb28e4b2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:18:04.634 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:18:04.632+0000 7fb28e4b2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:18:04.689 INFO:teuthology.orchestra.run.smithi153.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-09-22T03:18:04.738 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-22T03:18:04.739 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-22T03:18:05.110 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 38 16 1832 1816 0.373306 3.05469 0.0499183 0.334672 2024-09-22T03:18:05.563 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [1] live_osds: [0, 2, 3] 2024-09-22T03:18:05.564 INFO:tasks.thrashosds.thrasher:Reviving osd 1 2024-09-22T03:18:05.564 INFO:tasks.ceph.osd.1:Restarting daemon 2024-09-22T03:18:05.564 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2>> /var/log/ceph/ceph-osd.1.log 2024-09-22T03:18:05.567 INFO:tasks.ceph.osd.1:Started 2024-09-22T03:18:05.567 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-22T03:18:05.679 INFO:teuthology.orchestra.run.smithi153.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-09-22T03:18:05.682 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-22T03:18:05.683 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2024-09-22T03:18:06.110 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 39 16 2189 2173 0.43524 2.78906 0.0584602 0.287014 2024-09-22T03:18:06.739 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-22T03:18:06.849 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:18:06.846+0000 7f9ba9ad7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:18:06.852 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:18:06.851+0000 7f9ba9ad7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:18:07.021 INFO:teuthology.orchestra.run.smithi153.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-09-22T03:18:07.059 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-22T03:18:07.059 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-22T03:18:07.110 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:2024-09-22T03:18:07.109574+0000 min lat: 0.00665283 max lat: 33.1634 avg lat: 0.268705 2024-09-22T03:18:07.110 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-22T03:18:07.110 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 40 16 2393 2377 0.464198 1.59375 0.0491881 0.268705 2024-09-22T03:18:08.110 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 41 16 2770 2754 0.524704 2.94531 0.0370598 0.238094 2024-09-22T03:18:09.060 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-22T03:18:09.110 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 42 16 3131 3115 0.579353 2.82031 0.0213865 0.215571 2024-09-22T03:18:09.200 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:18:09.196+0000 7f8269863640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:18:09.204 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:18:09.202+0000 7f8269863640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:18:09.222 INFO:teuthology.orchestra.run.smithi153.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-09-22T03:18:09.280 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-22T03:18:09.280 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-22T03:18:10.110 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 43 16 3468 3452 0.627101 2.63281 0.0641737 0.199244 2024-09-22T03:18:10.684 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-22T03:18:10.807 INFO:teuthology.orchestra.run.smithi153.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-09-22T03:18:10.811 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-22T03:18:10.811 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2024-09-22T03:18:11.111 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 44 16 3850 3834 0.680668 2.98438 0.0460299 0.183532 2024-09-22T03:18:11.281 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-22T03:18:11.421 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:18:11.418+0000 7f5e4606c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:18:11.424 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:18:11.423+0000 7f5e4606c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:18:11.442 INFO:teuthology.orchestra.run.smithi153.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-09-22T03:18:11.563 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-22T03:18:11.564 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-22T03:18:12.110 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 45 16 4170 4154 0.721092 2.5 0.0132465 0.173252 2024-09-22T03:18:13.110 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 46 16 4536 4520 0.767569 2.85938 0.0365554 0.16277 2024-09-22T03:18:13.565 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-22T03:18:13.687 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:18:13.682+0000 7fe374197640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:18:13.692 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:18:13.690+0000 7fe374197640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:18:13.779 INFO:teuthology.orchestra.run.smithi153.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-09-22T03:18:13.875 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-22T03:18:13.875 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-22T03:18:14.111 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 47 16 4885 4869 0.809243 2.72656 0.0379043 0.154382 2024-09-22T03:18:15.111 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 48 16 5199 5183 0.843486 2.45312 0.0618813 0.148106 2024-09-22T03:18:15.812 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-22T03:18:15.876 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-22T03:18:15.957 INFO:teuthology.orchestra.run.smithi153.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-09-22T03:18:15.960 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-22T03:18:15.961 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2024-09-22T03:18:15.999 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:18:15.996+0000 7f2d08a2c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:18:16.003 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:18:16.001+0000 7f2d08a2c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:18:16.111 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 49 16 5528 5512 0.878721 2.57031 0.0892267 0.142122 2024-09-22T03:18:16.126 INFO:teuthology.orchestra.run.smithi153.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-09-22T03:18:16.215 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-22T03:18:16.215 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-22T03:18:17.111 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 50 16 5830 5814 0.90833 2.35938 0.0838043 0.13729 2024-09-22T03:18:18.111 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 51 16 5830 5814 0.89052 0 - 0.13729 2024-09-22T03:18:18.216 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-22T03:18:18.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:18:18.315+0000 7f7eaec2c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:18:18.321 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:18:18.319+0000 7f7eaec2c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:18:19.112 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 52 16 5830 5814 0.873395 0 - 0.13729 2024-09-22T03:18:20.111 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 53 16 6025 6009 0.885657 0.507812 0.0312487 0.141043 2024-09-22T03:18:20.387 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-22T03:18:20.536 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:18:20.532+0000 7f12e5b97640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:18:20.542 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:18:20.540+0000 7f12e5b97640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:18:20.962 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-22T03:18:21.103 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-22T03:18:21.104 INFO:teuthology.orchestra.run.smithi153.stdout: "ops_in_flight": [ 2024-09-22T03:18:21.104 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-22T03:18:21.104 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "background_recovery_sub", 2024-09-22T03:18:21.104 INFO:teuthology.orchestra.run.smithi153.stdout: "id": 16777424, 2024-09-22T03:18:21.104 INFO:teuthology.orchestra.run.smithi153.stdout: "detail": {} 2024-09-22T03:18:21.104 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:18:21.104 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-22T03:18:21.104 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "replicated_request", 2024-09-22T03:18:21.104 INFO:teuthology.orchestra.run.smithi153.stdout: "id": 503, 2024-09-22T03:18:21.104 INFO:teuthology.orchestra.run.smithi153.stdout: "detail": { 2024-09-22T03:18:21.104 INFO:teuthology.orchestra.run.smithi153.stdout: "RepRequest": { 2024-09-22T03:18:21.104 INFO:teuthology.orchestra.run.smithi153.stdout: "reqid": "client.4977.0:6239", 2024-09-22T03:18:21.104 INFO:teuthology.orchestra.run.smithi153.stdout: "pgid": "8.c", 2024-09-22T03:18:21.105 INFO:teuthology.orchestra.run.smithi153.stdout: "map_epoch": 124, 2024-09-22T03:18:21.105 INFO:teuthology.orchestra.run.smithi153.stdout: "min_epoch": 122, 2024-09-22T03:18:21.105 INFO:teuthology.orchestra.run.smithi153.stdout: "oid": "8:37a33845:::benchmark_data_smithi153_41696_object779:head", 2024-09-22T03:18:21.105 INFO:teuthology.orchestra.run.smithi153.stdout: "from": "0" 2024-09-22T03:18:21.105 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:18:21.105 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:18:21.105 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:18:21.105 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-22T03:18:21.105 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-22T03:18:21.111 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 54 16 6251 6235 0.901949 1.76562 0.122922 0.138379 2024-09-22T03:18:21.117 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-09-22T03:18:21.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:18:21.243+0000 7f4c3390e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:18:21.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:18:21.259+0000 7f4c3390e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:18:22.038 INFO:teuthology.orchestra.run.smithi153.stderr:reweighted osd.0 to 1 (10000) 2024-09-22T03:18:22.059 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-09-22T03:18:22.111 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 55 16 6296 6280 0.891942 0.351562 0.029249 0.137715 2024-09-22T03:18:22.189 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:18:22.181+0000 7f9dc8e6c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:18:22.193 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:18:22.191+0000 7f9dc8e6c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:18:22.697 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-22T03:18:22.827 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:18:22.823+0000 7f74f3997640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:18:22.830 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:18:22.829+0000 7f74f3997640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:18:23.045 INFO:teuthology.orchestra.run.smithi153.stderr:reweighted osd.1 to 1 (10000) 2024-09-22T03:18:23.062 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-09-22T03:18:23.111 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 56 16 6345 6329 0.88285 0.382812 0.719966 0.139789 2024-09-22T03:18:23.206 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:18:23.196+0000 7f496f317640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:18:23.210 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:18:23.209+0000 7f496f317640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:18:23.927 INFO:teuthology.orchestra.run.smithi153.stderr:reweighted osd.2 to 1 (10000) 2024-09-22T03:18:23.945 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-09-22T03:18:24.099 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:18:24.091+0000 7fa36b397640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:18:24.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:18:24.102+0000 7fa36b397640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:18:24.111 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 57 16 6432 6416 0.879285 0.679688 0.0387916 0.142095 2024-09-22T03:18:24.932 INFO:teuthology.orchestra.run.smithi153.stderr:reweighted osd.3 to 1 (10000) 2024-09-22T03:18:24.951 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-09-22T03:18:24.951 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-22T03:18:24.959 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-22T03:18:25.088 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:18:25.082+0000 7fb66846c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:18:25.092 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:18:25.091+0000 7fb66846c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:18:25.111 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 58 16 6453 6437 0.866953 0.164062 0.0484521 0.143505 2024-09-22T03:18:25.129 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:18:25.125+0000 7f1b5806c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:18:25.133 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:18:25.131+0000 7f1b5806c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:18:25.354 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-22T03:18:25.355 INFO:teuthology.orchestra.run.smithi153.stderr:dumped all 2024-09-22T03:18:25.376 INFO:teuthology.orchestra.run.smithi153.stdout:{"pg_ready":true,"pg_map":{"version":432,"stamp":"2024-09-22T03:18:24.931954+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":51782195,"num_objects":786,"num_object_clones":0,"num_object_copies":2358,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":370,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":6388,"num_write_kb":51888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":259,"num_bytes_recovered":15966208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6343,"ondisk_log_size":6343,"up":75,"acting":63,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":223848,"kb_used_data":60136,"kb_used_omap":0,"kb_used_meta":163648,"kb_avail":377263512,"statfs":{"total":386547056640,"available":386317836288,"internally_reserved":0,"allocated":61579264,"data_stored":142254820,"data_compressed":7147069,"data_compressed_allocated":57352192,"data_compressed_original":140533760,"omap_allocated":0,"internal_metadata":167575552},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":3801088,"num_objects":58,"num_object_clones":0,"num_object_copies":174,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-246,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":464,"num_write_kb":3712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":216,"num_bytes_recovered":14065664,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.895205"},"pg_stats":[{"pgid":"8.7","version":"124'504","reported_seq":630,"reported_epoch":124,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-22T03:18:21.674252+0000","last_change":"2024-09-22T03:18:21.164281+0000","last_active":"2024-09-22T03:18:21.674252+0000","last_peered":"2024-09-22T03:18:21.674252+0000","last_clean":"2024-09-22T03:17:25.017919+0000","last_became_active":"2024-09-22T03:18:19.397637+0000","last_became_peered":"2024-09-22T03:18:19.397637+0000","last_unstale":"2024-09-22T03:18:21.674252+0000","last_undegraded":"2024-09-22T03:18:19.061412+0000","last_fullsized":"2024-09-22T03:18:19.053844+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:17:25.017919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:17:25.017919+0000","last_clean_scrub_stamp":"2024-09-22T03:17:25.017919+0000","objects_scrubbed":0,"log_size":504,"log_dups_size":0,"ondisk_log_size":504,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4128768,"num_objects":63,"num_object_clones":0,"num_object_copies":189,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":504,"num_write_kb":4032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":2883584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":48}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.6","version":"124'352","reported_seq":460,"reported_epoch":124,"state":"active+undersized+remapped","last_fresh":"2024-09-22T03:18:21.397439+0000","last_change":"2024-09-22T03:18:21.398718+0000","last_active":"2024-09-22T03:18:21.397439+0000","last_peered":"2024-09-22T03:18:21.397439+0000","last_clean":"2024-09-22T03:17:28.360944+0000","last_became_active":"2024-09-22T03:18:19.318455+0000","last_became_peered":"2024-09-22T03:18:19.318455+0000","last_unstale":"2024-09-22T03:18:21.397439+0000","last_undegraded":"2024-09-22T03:18:19.051817+0000","last_fullsized":"2024-09-22T03:18:19.036657+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:17:25.017919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:17:25.017919+0000","last_clean_scrub_stamp":"2024-09-22T03:17:25.017919+0000","objects_scrubbed":0,"log_size":352,"log_dups_size":0,"ondisk_log_size":352,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2883584,"num_objects":44,"num_object_clones":0,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":352,"num_write_kb":2816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":2260992,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0],"avail_no_missing":["2","0","1"],"object_location_counts":[{"shards":"0,2","objects":44}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.5","version":"123'432","reported_seq":481,"reported_epoch":124,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-22T03:18:21.913645+0000","last_change":"2024-09-22T03:18:21.832176+0000","last_active":"2024-09-22T03:18:21.913645+0000","last_peered":"2024-09-22T03:18:21.913645+0000","last_clean":"2024-09-22T03:17:28.351270+0000","last_became_active":"2024-09-22T03:18:19.319765+0000","last_became_peered":"2024-09-22T03:18:19.319765+0000","last_unstale":"2024-09-22T03:18:21.913645+0000","last_undegraded":"2024-09-22T03:18:19.107317+0000","last_fullsized":"2024-09-22T03:18:19.094542+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:17:25.017919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:17:25.017919+0000","last_clean_scrub_stamp":"2024-09-22T03:17:25.017919+0000","objects_scrubbed":0,"log_size":432,"log_dups_size":0,"ondisk_log_size":432,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3538944,"num_objects":54,"num_object_clones":0,"num_object_copies":162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":37,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":432,"num_write_kb":3456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":458752,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,2","objects":17}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.4","version":"123'464","reported_seq":605,"reported_epoch":124,"state":"active+undersized+remapped","last_fresh":"2024-09-22T03:18:21.118315+0000","last_change":"2024-09-22T03:18:21.120223+0000","last_active":"2024-09-22T03:18:21.118315+0000","last_peered":"2024-09-22T03:18:21.118315+0000","last_clean":"2024-09-22T03:17:25.017919+0000","last_became_active":"2024-09-22T03:18:19.253832+0000","last_became_peered":"2024-09-22T03:18:19.253832+0000","last_unstale":"2024-09-22T03:18:21.118315+0000","last_undegraded":"2024-09-22T03:18:19.080228+0000","last_fullsized":"2024-09-22T03:18:19.066463+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:17:25.017919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:17:25.017919+0000","last_clean_scrub_stamp":"2024-09-22T03:17:25.017919+0000","objects_scrubbed":0,"log_size":464,"log_dups_size":0,"ondisk_log_size":464,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3801088,"num_objects":58,"num_object_clones":0,"num_object_copies":174,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":464,"num_write_kb":3712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":51,"num_bytes_recovered":3342336,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,3],"avail_no_missing":["0","1","3"],"object_location_counts":[{"shards":"0,3","objects":58}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.3","version":"123'328","reported_seq":325,"reported_epoch":124,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-22T03:18:21.080958+0000","last_change":"2024-09-22T03:18:19.414081+0000","last_active":"2024-09-22T03:18:21.080958+0000","last_peered":"2024-09-22T03:18:21.080958+0000","last_clean":"2024-09-22T03:17:25.017919+0000","last_became_active":"2024-09-22T03:18:19.409234+0000","last_became_peered":"2024-09-22T03:18:19.409234+0000","last_unstale":"2024-09-22T03:18:21.080958+0000","last_undegraded":"2024-09-22T03:18:19.072687+0000","last_fullsized":"2024-09-22T03:18:19.066510+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:17:25.017919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:17:25.017919+0000","last_clean_scrub_stamp":"2024-09-22T03:17:25.017919+0000","objects_scrubbed":0,"log_size":328,"log_dups_size":0,"ondisk_log_size":328,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2686976,"num_objects":41,"num_object_clones":0,"num_object_copies":123,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":328,"num_write_kb":2624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":5}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.2","version":"123'432","reported_seq":483,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-22T03:18:21.081501+0000","last_change":"2024-09-22T03:18:01.847644+0000","last_active":"2024-09-22T03:18:21.081501+0000","last_peered":"2024-09-22T03:18:21.081501+0000","last_clean":"2024-09-22T03:18:21.081501+0000","last_became_active":"2024-09-22T03:18:01.225010+0000","last_became_peered":"2024-09-22T03:18:01.225010+0000","last_unstale":"2024-09-22T03:18:21.081501+0000","last_undegraded":"2024-09-22T03:18:21.081501+0000","last_fullsized":"2024-09-22T03:18:21.081501+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:17:25.017919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:17:25.017919+0000","last_clean_scrub_stamp":"2024-09-22T03:17:25.017919+0000","objects_scrubbed":0,"log_size":432,"log_dups_size":0,"ondisk_log_size":432,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3538944,"num_objects":54,"num_object_clones":0,"num_object_copies":162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":432,"num_write_kb":3456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":393216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1","version":"123'463","reported_seq":454,"reported_epoch":123,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-22T03:18:19.748831+0000","last_change":"2024-09-22T03:18:19.402090+0000","last_active":"2024-09-22T03:18:19.748831+0000","last_peered":"2024-09-22T03:18:19.748831+0000","last_clean":"2024-09-22T03:17:25.017919+0000","last_became_active":"2024-09-22T03:18:19.398712+0000","last_became_peered":"2024-09-22T03:18:19.398712+0000","last_unstale":"2024-09-22T03:18:19.748831+0000","last_undegraded":"2024-09-22T03:18:19.055325+0000","last_fullsized":"2024-09-22T03:18:19.047603+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:17:25.017919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:17:25.017919+0000","last_clean_scrub_stamp":"2024-09-22T03:17:25.017919+0000","objects_scrubbed":0,"log_size":463,"log_dups_size":0,"ondisk_log_size":463,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3801088,"num_objects":58,"num_object_clones":0,"num_object_copies":174,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":51,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":464,"num_write_kb":3712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3","objects":7}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"123'344","reported_seq":429,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-22T03:18:22.111803+0000","last_change":"2024-09-22T03:18:22.112809+0000","last_active":"2024-09-22T03:18:22.111803+0000","last_peered":"2024-09-22T03:18:22.111803+0000","last_clean":"2024-09-22T03:17:25.017919+0000","last_became_active":"2024-09-22T03:18:22.107439+0000","last_became_peered":"2024-09-22T03:18:22.107439+0000","last_unstale":"2024-09-22T03:18:22.111803+0000","last_undegraded":"2024-09-22T03:18:22.111803+0000","last_fullsized":"2024-09-22T03:18:22.111803+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:17:25.017919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:17:25.017919+0000","last_clean_scrub_stamp":"2024-09-22T03:17:25.017919+0000","objects_scrubbed":0,"log_size":344,"log_dups_size":0,"ondisk_log_size":344,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2818048,"num_objects":43,"num_object_clones":0,"num_object_copies":129,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":344,"num_write_kb":2752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":2490368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.9","version":"124'439","reported_seq":488,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-22T03:18:21.170976+0000","last_change":"2024-09-22T03:18:01.629983+0000","last_active":"2024-09-22T03:18:21.170976+0000","last_peered":"2024-09-22T03:18:21.170976+0000","last_clean":"2024-09-22T03:18:21.170976+0000","last_became_active":"2024-09-22T03:18:01.196590+0000","last_became_peered":"2024-09-22T03:18:01.196590+0000","last_unstale":"2024-09-22T03:18:21.170976+0000","last_undegraded":"2024-09-22T03:18:21.170976+0000","last_fullsized":"2024-09-22T03:18:21.170976+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:17:25.017919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:17:25.017919+0000","last_clean_scrub_stamp":"2024-09-22T03:17:25.017919+0000","objects_scrubbed":0,"log_size":439,"log_dups_size":0,"ondisk_log_size":439,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3604480,"num_objects":55,"num_object_clones":0,"num_object_copies":165,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":440,"num_write_kb":3520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":393216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":161,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-09-22T03:18:19.056022+0000","last_change":"2024-09-22T03:18:17.050118+0000","last_active":"2024-09-22T03:18:19.056022+0000","last_peered":"2024-09-22T03:18:19.056022+0000","last_clean":"2024-09-22T03:18:19.056022+0000","last_became_active":"2024-09-22T03:18:01.415712+0000","last_became_peered":"2024-09-22T03:18:01.415712+0000","last_unstale":"2024-09-22T03:18:19.056022+0000","last_undegraded":"2024-09-22T03:18:19.056022+0000","last_fullsized":"2024-09-22T03:18:19.056022+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.8","version":"125'384","reported_seq":524,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-22T03:18:22.143102+0000","last_change":"2024-09-22T03:18:22.093165+0000","last_active":"2024-09-22T03:18:22.143102+0000","last_peered":"2024-09-22T03:18:22.143102+0000","last_clean":"2024-09-22T03:18:22.143102+0000","last_became_active":"2024-09-22T03:18:22.084001+0000","last_became_peered":"2024-09-22T03:18:22.084001+0000","last_unstale":"2024-09-22T03:18:22.143102+0000","last_undegraded":"2024-09-22T03:18:22.143102+0000","last_fullsized":"2024-09-22T03:18:22.143102+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:17:25.017919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:17:25.017919+0000","last_clean_scrub_stamp":"2024-09-22T03:17:25.017919+0000","objects_scrubbed":0,"log_size":384,"log_dups_size":0,"ondisk_log_size":384,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3153920,"num_objects":49,"num_object_clones":0,"num_object_copies":147,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":385,"num_write_kb":3080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":2957312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":296,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-22T03:18:21.077512+0000","last_change":"2024-09-22T03:18:19.078674+0000","last_active":"2024-09-22T03:18:21.077512+0000","last_peered":"2024-09-22T03:18:21.077512+0000","last_clean":"2024-09-22T03:18:21.077512+0000","last_became_active":"2024-09-22T03:18:18.121868+0000","last_became_peered":"2024-09-22T03:18:18.121868+0000","last_unstale":"2024-09-22T03:18:21.077512+0000","last_undegraded":"2024-09-22T03:18:21.077512+0000","last_fullsized":"2024-09-22T03:18:21.077512+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.b","version":"123'296","reported_seq":344,"reported_epoch":124,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-22T03:18:21.908775+0000","last_change":"2024-09-22T03:18:21.839090+0000","last_active":"2024-09-22T03:18:21.908775+0000","last_peered":"2024-09-22T03:18:21.908775+0000","last_clean":"2024-09-22T03:17:28.441691+0000","last_became_active":"2024-09-22T03:18:19.322920+0000","last_became_peered":"2024-09-22T03:18:19.322920+0000","last_unstale":"2024-09-22T03:18:21.908775+0000","last_undegraded":"2024-09-22T03:18:19.116974+0000","last_fullsized":"2024-09-22T03:18:19.109309+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:17:25.017919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:17:25.017919+0000","last_clean_scrub_stamp":"2024-09-22T03:17:25.017919+0000","objects_scrubbed":0,"log_size":296,"log_dups_size":0,"ondisk_log_size":296,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2424832,"num_objects":37,"num_object_clones":0,"num_object_copies":111,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":296,"num_write_kb":2368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":393216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,2","objects":12}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":293,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-22T03:18:21.047269+0000","last_change":"2024-09-22T03:18:19.067090+0000","last_active":"2024-09-22T03:18:21.047269+0000","last_peered":"2024-09-22T03:18:21.047269+0000","last_clean":"2024-09-22T03:18:21.047269+0000","last_became_active":"2024-09-22T03:18:18.104717+0000","last_became_peered":"2024-09-22T03:18:18.104717+0000","last_unstale":"2024-09-22T03:18:21.047269+0000","last_undegraded":"2024-09-22T03:18:21.047269+0000","last_fullsized":"2024-09-22T03:18:21.047269+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.a","version":"123'311","reported_seq":322,"reported_epoch":123,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-22T03:18:20.074408+0000","last_change":"2024-09-22T03:18:19.400988+0000","last_active":"2024-09-22T03:18:20.074408+0000","last_peered":"2024-09-22T03:18:20.074408+0000","last_clean":"2024-09-22T03:17:25.017919+0000","last_became_active":"2024-09-22T03:18:19.395317+0000","last_became_peered":"2024-09-22T03:18:19.395317+0000","last_unstale":"2024-09-22T03:18:20.074408+0000","last_undegraded":"2024-09-22T03:18:19.062506+0000","last_fullsized":"2024-09-22T03:18:19.057039+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:17:25.017919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:17:25.017919+0000","last_clean_scrub_stamp":"2024-09-22T03:17:25.017919+0000","objects_scrubbed":0,"log_size":311,"log_dups_size":0,"ondisk_log_size":311,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2555904,"num_objects":39,"num_object_clones":0,"num_object_copies":117,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":312,"num_write_kb":2496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":183,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-09-22T03:18:19.046544+0000","last_change":"2024-09-22T03:18:19.050753+0000","last_active":"2024-09-22T03:18:19.046544+0000","last_peered":"2024-09-22T03:18:19.046544+0000","last_clean":"2024-09-22T03:18:19.046544+0000","last_became_active":"2024-09-22T03:18:18.099051+0000","last_became_peered":"2024-09-22T03:18:18.099051+0000","last_unstale":"2024-09-22T03:18:19.046544+0000","last_undegraded":"2024-09-22T03:18:19.046544+0000","last_fullsized":"2024-09-22T03:18:19.046544+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"12'97","reported_seq":417,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-22T03:18:21.080339+0000","last_change":"2024-09-22T03:18:18.126160+0000","last_active":"2024-09-22T03:18:21.080339+0000","last_peered":"2024-09-22T03:18:21.080339+0000","last_clean":"2024-09-22T03:18:21.080339+0000","last_became_active":"2024-09-22T03:18:18.122292+0000","last_became_peered":"2024-09-22T03:18:18.122292+0000","last_unstale":"2024-09-22T03:18:21.080339+0000","last_undegraded":"2024-09-22T03:18:21.080339+0000","last_fullsized":"2024-09-22T03:18:21.080339+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.d","version":"123'320","reported_seq":357,"reported_epoch":124,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-22T03:18:21.081962+0000","last_change":"2024-09-22T03:18:19.412042+0000","last_active":"2024-09-22T03:18:21.081962+0000","last_peered":"2024-09-22T03:18:21.081962+0000","last_clean":"2024-09-22T03:17:28.160849+0000","last_became_active":"2024-09-22T03:18:19.404230+0000","last_became_peered":"2024-09-22T03:18:19.404230+0000","last_unstale":"2024-09-22T03:18:21.081962+0000","last_undegraded":"2024-09-22T03:18:19.088117+0000","last_fullsized":"2024-09-22T03:18:19.081933+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:17:25.017919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:17:25.017919+0000","last_clean_scrub_stamp":"2024-09-22T03:17:25.017919+0000","objects_scrubbed":0,"log_size":320,"log_dups_size":0,"ondisk_log_size":320,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2621440,"num_objects":40,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":320,"num_write_kb":2560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,2","objects":5}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":249,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-22T03:18:22.060900+0000","last_change":"2024-09-22T03:18:19.044456+0000","last_active":"2024-09-22T03:18:22.060900+0000","last_peered":"2024-09-22T03:18:22.060900+0000","last_clean":"2024-09-22T03:18:22.060900+0000","last_became_active":"2024-09-22T03:18:18.129858+0000","last_became_peered":"2024-09-22T03:18:18.129858+0000","last_unstale":"2024-09-22T03:18:22.060900+0000","last_undegraded":"2024-09-22T03:18:22.060900+0000","last_fullsized":"2024-09-22T03:18:22.060900+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.c","version":"124'431","reported_seq":473,"reported_epoch":124,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-22T03:18:21.083127+0000","last_change":"2024-09-22T03:18:19.404282+0000","last_active":"2024-09-22T03:18:21.083127+0000","last_peered":"2024-09-22T03:18:21.083127+0000","last_clean":"2024-09-22T03:17:25.017919+0000","last_became_active":"2024-09-22T03:18:19.398884+0000","last_became_peered":"2024-09-22T03:18:19.398884+0000","last_unstale":"2024-09-22T03:18:21.083127+0000","last_undegraded":"2024-09-22T03:18:19.061071+0000","last_fullsized":"2024-09-22T03:18:19.052000+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:17:25.017919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:17:25.017919+0000","last_clean_scrub_stamp":"2024-09-22T03:17:25.017919+0000","objects_scrubbed":0,"log_size":431,"log_dups_size":0,"ondisk_log_size":431,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3538944,"num_objects":54,"num_object_clones":0,"num_object_copies":162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":49,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":432,"num_write_kb":3456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":5}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":268,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-22T03:18:21.047791+0000","last_change":"2024-09-22T03:18:17.081823+0000","last_active":"2024-09-22T03:18:21.047791+0000","last_peered":"2024-09-22T03:18:21.047791+0000","last_clean":"2024-09-22T03:18:21.047791+0000","last_became_active":"2024-09-22T03:18:01.232732+0000","last_became_peered":"2024-09-22T03:18:01.232732+0000","last_unstale":"2024-09-22T03:18:21.047791+0000","last_undegraded":"2024-09-22T03:18:21.047791+0000","last_fullsized":"2024-09-22T03:18:21.047791+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.f","version":"123'391","reported_seq":406,"reported_epoch":123,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-22T03:18:19.682193+0000","last_change":"2024-09-22T03:18:19.394293+0000","last_active":"2024-09-22T03:18:19.682193+0000","last_peered":"2024-09-22T03:18:19.682193+0000","last_clean":"2024-09-22T03:17:25.017919+0000","last_became_active":"2024-09-22T03:18:19.391348+0000","last_became_peered":"2024-09-22T03:18:19.391348+0000","last_unstale":"2024-09-22T03:18:19.682193+0000","last_undegraded":"2024-09-22T03:18:19.043600+0000","last_fullsized":"2024-09-22T03:18:19.036283+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:17:25.017919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:17:25.017919+0000","last_clean_scrub_stamp":"2024-09-22T03:17:25.017919+0000","objects_scrubbed":0,"log_size":391,"log_dups_size":0,"ondisk_log_size":391,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3211264,"num_objects":49,"num_object_clones":0,"num_object_copies":147,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":46,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":392,"num_write_kb":3136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":297,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-22T03:18:22.047685+0000","last_change":"2024-09-22T03:18:19.043166+0000","last_active":"2024-09-22T03:18:22.047685+0000","last_peered":"2024-09-22T03:18:22.047685+0000","last_clean":"2024-09-22T03:18:22.047685+0000","last_became_active":"2024-09-22T03:18:18.098084+0000","last_became_peered":"2024-09-22T03:18:18.098084+0000","last_unstale":"2024-09-22T03:18:22.047685+0000","last_undegraded":"2024-09-22T03:18:22.047685+0000","last_fullsized":"2024-09-22T03:18:22.047685+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.e","version":"123'352","reported_seq":393,"reported_epoch":124,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-22T03:18:21.079821+0000","last_change":"2024-09-22T03:18:19.436894+0000","last_active":"2024-09-22T03:18:21.079821+0000","last_peered":"2024-09-22T03:18:21.079821+0000","last_clean":"2024-09-22T03:17:25.017919+0000","last_became_active":"2024-09-22T03:18:19.428685+0000","last_became_peered":"2024-09-22T03:18:19.428685+0000","last_unstale":"2024-09-22T03:18:21.079821+0000","last_undegraded":"2024-09-22T03:18:19.093276+0000","last_fullsized":"2024-09-22T03:18:19.081651+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:17:25.017919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:17:25.017919+0000","last_clean_scrub_stamp":"2024-09-22T03:17:25.017919+0000","objects_scrubbed":0,"log_size":352,"log_dups_size":0,"ondisk_log_size":352,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2883584,"num_objects":44,"num_object_clones":0,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":40,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":352,"num_write_kb":2816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":4}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":290,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-22T03:18:22.050428+0000","last_change":"2024-09-22T03:18:19.042797+0000","last_active":"2024-09-22T03:18:22.050428+0000","last_peered":"2024-09-22T03:18:22.050428+0000","last_clean":"2024-09-22T03:18:22.050428+0000","last_became_active":"2024-09-22T03:18:18.118143+0000","last_became_peered":"2024-09-22T03:18:18.118143+0000","last_unstale":"2024-09-22T03:18:22.050428+0000","last_undegraded":"2024-09-22T03:18:22.050428+0000","last_fullsized":"2024-09-22T03:18:22.050428+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":51191808,"num_objects":782,"num_object_clones":0,"num_object_copies":2346,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":370,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6249,"num_write_kb":49992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":259,"num_bytes_recovered":15966208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":57761792,"data_stored":140394496,"data_compressed":7136536,"data_compressed_allocated":57761792,"data_compressed_original":140394496,"omap_allocated":0,"internal_metadata":0},"log_size":6243,"ondisk_log_size":6243,"up":48,"acting":36,"num_store_stats":4},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1180736,"data_compressed":59924,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":119,"seq":511101108244,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":49328,"kb_used_data":18400,"kb_used_omap":0,"kb_used_meta":30912,"kb_avail":94322512,"statfs":{"total":96636764160,"available":96586252288,"internally_reserved":0,"allocated":18841600,"data_stored":38433911,"data_compressed":1951998,"data_compressed_allocated":17805312,"data_compressed_original":38019072,"omap_allocated":0,"internal_metadata":31653888},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":100,"seq":429496729724,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":55904,"kb_used_data":14992,"kb_used_omap":0,"kb_used_meta":40896,"kb_avail":94315936,"statfs":{"total":96636764160,"available":96579518464,"internally_reserved":0,"allocated":15351808,"data_stored":29513367,"data_compressed":1501350,"data_compressed_allocated":14295040,"data_compressed_original":29081600,"omap_allocated":0,"internal_metadata":41877504},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":121,"seq":519691042821,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":43040,"kb_used_data":5520,"kb_used_omap":0,"kb_used_meta":37504,"kb_avail":94328800,"statfs":{"total":96636764160,"available":96592691200,"internally_reserved":0,"allocated":5652480,"data_stored":32011011,"data_compressed":1532892,"data_compressed_allocated":4579328,"data_compressed_original":31571968,"omap_allocated":0,"internal_metadata":38404096},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":73,"seq":313532612833,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":75576,"kb_used_data":21224,"kb_used_omap":0,"kb_used_meta":54336,"kb_avail":94296264,"statfs":{"total":96636764160,"available":96559374336,"internally_reserved":0,"allocated":21733376,"data_stored":42296531,"data_compressed":2160829,"data_compressed_allocated":20672512,"data_compressed_original":41861120,"omap_allocated":0,"internal_metadata":55640064},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29962,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29962,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20631552,"data_stored":41287680,"data_compressed":2130867,"data_compressed_allocated":20631552,"data_compressed_original":41287680,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4612096,"data_stored":31662080,"data_compressed":1534907,"data_compressed_allocated":4612096,"data_compressed_original":31662080,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":14385152,"data_stored":28770304,"data_compressed":1484924,"data_compressed_allocated":14385152,"data_compressed_original":28770304,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":18132992,"data_stored":38674432,"data_compressed":1985838,"data_compressed_allocated":18132992,"data_compressed_original":38674432,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-22T03:18:25.377 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-22T03:18:25.513 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:18:25.506+0000 7f0859597640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:18:25.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:18:25.515+0000 7f0859597640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:18:25.769 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-22T03:18:25.769 INFO:teuthology.orchestra.run.smithi153.stderr:dumped all 2024-09-22T03:18:25.785 INFO:teuthology.orchestra.run.smithi153.stdout:{"pg_ready":true,"pg_map":{"version":432,"stamp":"2024-09-22T03:18:24.931954+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":51782195,"num_objects":786,"num_object_clones":0,"num_object_copies":2358,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":370,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":6388,"num_write_kb":51888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":259,"num_bytes_recovered":15966208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6343,"ondisk_log_size":6343,"up":75,"acting":63,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":223848,"kb_used_data":60136,"kb_used_omap":0,"kb_used_meta":163648,"kb_avail":377263512,"statfs":{"total":386547056640,"available":386317836288,"internally_reserved":0,"allocated":61579264,"data_stored":142254820,"data_compressed":7147069,"data_compressed_allocated":57352192,"data_compressed_original":140533760,"omap_allocated":0,"internal_metadata":167575552},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":3801088,"num_objects":58,"num_object_clones":0,"num_object_copies":174,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-246,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":464,"num_write_kb":3712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":216,"num_bytes_recovered":14065664,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.895205"},"pg_stats":[{"pgid":"8.7","version":"124'504","reported_seq":630,"reported_epoch":124,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-22T03:18:21.674252+0000","last_change":"2024-09-22T03:18:21.164281+0000","last_active":"2024-09-22T03:18:21.674252+0000","last_peered":"2024-09-22T03:18:21.674252+0000","last_clean":"2024-09-22T03:17:25.017919+0000","last_became_active":"2024-09-22T03:18:19.397637+0000","last_became_peered":"2024-09-22T03:18:19.397637+0000","last_unstale":"2024-09-22T03:18:21.674252+0000","last_undegraded":"2024-09-22T03:18:19.061412+0000","last_fullsized":"2024-09-22T03:18:19.053844+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:17:25.017919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:17:25.017919+0000","last_clean_scrub_stamp":"2024-09-22T03:17:25.017919+0000","objects_scrubbed":0,"log_size":504,"log_dups_size":0,"ondisk_log_size":504,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4128768,"num_objects":63,"num_object_clones":0,"num_object_copies":189,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":504,"num_write_kb":4032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":2883584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":48}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.6","version":"124'352","reported_seq":460,"reported_epoch":124,"state":"active+undersized+remapped","last_fresh":"2024-09-22T03:18:21.397439+0000","last_change":"2024-09-22T03:18:21.398718+0000","last_active":"2024-09-22T03:18:21.397439+0000","last_peered":"2024-09-22T03:18:21.397439+0000","last_clean":"2024-09-22T03:17:28.360944+0000","last_became_active":"2024-09-22T03:18:19.318455+0000","last_became_peered":"2024-09-22T03:18:19.318455+0000","last_unstale":"2024-09-22T03:18:21.397439+0000","last_undegraded":"2024-09-22T03:18:19.051817+0000","last_fullsized":"2024-09-22T03:18:19.036657+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:17:25.017919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:17:25.017919+0000","last_clean_scrub_stamp":"2024-09-22T03:17:25.017919+0000","objects_scrubbed":0,"log_size":352,"log_dups_size":0,"ondisk_log_size":352,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2883584,"num_objects":44,"num_object_clones":0,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":352,"num_write_kb":2816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":2260992,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0],"avail_no_missing":["2","0","1"],"object_location_counts":[{"shards":"0,2","objects":44}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.5","version":"123'432","reported_seq":481,"reported_epoch":124,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-22T03:18:21.913645+0000","last_change":"2024-09-22T03:18:21.832176+0000","last_active":"2024-09-22T03:18:21.913645+0000","last_peered":"2024-09-22T03:18:21.913645+0000","last_clean":"2024-09-22T03:17:28.351270+0000","last_became_active":"2024-09-22T03:18:19.319765+0000","last_became_peered":"2024-09-22T03:18:19.319765+0000","last_unstale":"2024-09-22T03:18:21.913645+0000","last_undegraded":"2024-09-22T03:18:19.107317+0000","last_fullsized":"2024-09-22T03:18:19.094542+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:17:25.017919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:17:25.017919+0000","last_clean_scrub_stamp":"2024-09-22T03:17:25.017919+0000","objects_scrubbed":0,"log_size":432,"log_dups_size":0,"ondisk_log_size":432,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3538944,"num_objects":54,"num_object_clones":0,"num_object_copies":162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":37,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":432,"num_write_kb":3456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":458752,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,2","objects":17}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.4","version":"123'464","reported_seq":605,"reported_epoch":124,"state":"active+undersized+remapped","last_fresh":"2024-09-22T03:18:21.118315+0000","last_change":"2024-09-22T03:18:21.120223+0000","last_active":"2024-09-22T03:18:21.118315+0000","last_peered":"2024-09-22T03:18:21.118315+0000","last_clean":"2024-09-22T03:17:25.017919+0000","last_became_active":"2024-09-22T03:18:19.253832+0000","last_became_peered":"2024-09-22T03:18:19.253832+0000","last_unstale":"2024-09-22T03:18:21.118315+0000","last_undegraded":"2024-09-22T03:18:19.080228+0000","last_fullsized":"2024-09-22T03:18:19.066463+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:17:25.017919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:17:25.017919+0000","last_clean_scrub_stamp":"2024-09-22T03:17:25.017919+0000","objects_scrubbed":0,"log_size":464,"log_dups_size":0,"ondisk_log_size":464,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3801088,"num_objects":58,"num_object_clones":0,"num_object_copies":174,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":464,"num_write_kb":3712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":51,"num_bytes_recovered":3342336,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,3],"avail_no_missing":["0","1","3"],"object_location_counts":[{"shards":"0,3","objects":58}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.3","version":"123'328","reported_seq":325,"reported_epoch":124,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-22T03:18:21.080958+0000","last_change":"2024-09-22T03:18:19.414081+0000","last_active":"2024-09-22T03:18:21.080958+0000","last_peered":"2024-09-22T03:18:21.080958+0000","last_clean":"2024-09-22T03:17:25.017919+0000","last_became_active":"2024-09-22T03:18:19.409234+0000","last_became_peered":"2024-09-22T03:18:19.409234+0000","last_unstale":"2024-09-22T03:18:21.080958+0000","last_undegraded":"2024-09-22T03:18:19.072687+0000","last_fullsized":"2024-09-22T03:18:19.066510+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:17:25.017919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:17:25.017919+0000","last_clean_scrub_stamp":"2024-09-22T03:17:25.017919+0000","objects_scrubbed":0,"log_size":328,"log_dups_size":0,"ondisk_log_size":328,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2686976,"num_objects":41,"num_object_clones":0,"num_object_copies":123,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":328,"num_write_kb":2624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":5}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.2","version":"123'432","reported_seq":483,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-22T03:18:21.081501+0000","last_change":"2024-09-22T03:18:01.847644+0000","last_active":"2024-09-22T03:18:21.081501+0000","last_peered":"2024-09-22T03:18:21.081501+0000","last_clean":"2024-09-22T03:18:21.081501+0000","last_became_active":"2024-09-22T03:18:01.225010+0000","last_became_peered":"2024-09-22T03:18:01.225010+0000","last_unstale":"2024-09-22T03:18:21.081501+0000","last_undegraded":"2024-09-22T03:18:21.081501+0000","last_fullsized":"2024-09-22T03:18:21.081501+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:17:25.017919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:17:25.017919+0000","last_clean_scrub_stamp":"2024-09-22T03:17:25.017919+0000","objects_scrubbed":0,"log_size":432,"log_dups_size":0,"ondisk_log_size":432,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3538944,"num_objects":54,"num_object_clones":0,"num_object_copies":162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":432,"num_write_kb":3456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":393216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1","version":"123'463","reported_seq":454,"reported_epoch":123,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-22T03:18:19.748831+0000","last_change":"2024-09-22T03:18:19.402090+0000","last_active":"2024-09-22T03:18:19.748831+0000","last_peered":"2024-09-22T03:18:19.748831+0000","last_clean":"2024-09-22T03:17:25.017919+0000","last_became_active":"2024-09-22T03:18:19.398712+0000","last_became_peered":"2024-09-22T03:18:19.398712+0000","last_unstale":"2024-09-22T03:18:19.748831+0000","last_undegraded":"2024-09-22T03:18:19.055325+0000","last_fullsized":"2024-09-22T03:18:19.047603+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:17:25.017919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:17:25.017919+0000","last_clean_scrub_stamp":"2024-09-22T03:17:25.017919+0000","objects_scrubbed":0,"log_size":463,"log_dups_size":0,"ondisk_log_size":463,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3801088,"num_objects":58,"num_object_clones":0,"num_object_copies":174,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":51,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":464,"num_write_kb":3712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3","objects":7}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"123'344","reported_seq":429,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-22T03:18:22.111803+0000","last_change":"2024-09-22T03:18:22.112809+0000","last_active":"2024-09-22T03:18:22.111803+0000","last_peered":"2024-09-22T03:18:22.111803+0000","last_clean":"2024-09-22T03:17:25.017919+0000","last_became_active":"2024-09-22T03:18:22.107439+0000","last_became_peered":"2024-09-22T03:18:22.107439+0000","last_unstale":"2024-09-22T03:18:22.111803+0000","last_undegraded":"2024-09-22T03:18:22.111803+0000","last_fullsized":"2024-09-22T03:18:22.111803+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:17:25.017919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:17:25.017919+0000","last_clean_scrub_stamp":"2024-09-22T03:17:25.017919+0000","objects_scrubbed":0,"log_size":344,"log_dups_size":0,"ondisk_log_size":344,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2818048,"num_objects":43,"num_object_clones":0,"num_object_copies":129,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":344,"num_write_kb":2752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":2490368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.9","version":"124'439","reported_seq":488,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-22T03:18:21.170976+0000","last_change":"2024-09-22T03:18:01.629983+0000","last_active":"2024-09-22T03:18:21.170976+0000","last_peered":"2024-09-22T03:18:21.170976+0000","last_clean":"2024-09-22T03:18:21.170976+0000","last_became_active":"2024-09-22T03:18:01.196590+0000","last_became_peered":"2024-09-22T03:18:01.196590+0000","last_unstale":"2024-09-22T03:18:21.170976+0000","last_undegraded":"2024-09-22T03:18:21.170976+0000","last_fullsized":"2024-09-22T03:18:21.170976+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:17:25.017919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:17:25.017919+0000","last_clean_scrub_stamp":"2024-09-22T03:17:25.017919+0000","objects_scrubbed":0,"log_size":439,"log_dups_size":0,"ondisk_log_size":439,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3604480,"num_objects":55,"num_object_clones":0,"num_object_copies":165,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":440,"num_write_kb":3520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":393216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":161,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-09-22T03:18:19.056022+0000","last_change":"2024-09-22T03:18:17.050118+0000","last_active":"2024-09-22T03:18:19.056022+0000","last_peered":"2024-09-22T03:18:19.056022+0000","last_clean":"2024-09-22T03:18:19.056022+0000","last_became_active":"2024-09-22T03:18:01.415712+0000","last_became_peered":"2024-09-22T03:18:01.415712+0000","last_unstale":"2024-09-22T03:18:19.056022+0000","last_undegraded":"2024-09-22T03:18:19.056022+0000","last_fullsized":"2024-09-22T03:18:19.056022+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.8","version":"125'384","reported_seq":524,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-22T03:18:22.143102+0000","last_change":"2024-09-22T03:18:22.093165+0000","last_active":"2024-09-22T03:18:22.143102+0000","last_peered":"2024-09-22T03:18:22.143102+0000","last_clean":"2024-09-22T03:18:22.143102+0000","last_became_active":"2024-09-22T03:18:22.084001+0000","last_became_peered":"2024-09-22T03:18:22.084001+0000","last_unstale":"2024-09-22T03:18:22.143102+0000","last_undegraded":"2024-09-22T03:18:22.143102+0000","last_fullsized":"2024-09-22T03:18:22.143102+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:17:25.017919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:17:25.017919+0000","last_clean_scrub_stamp":"2024-09-22T03:17:25.017919+0000","objects_scrubbed":0,"log_size":384,"log_dups_size":0,"ondisk_log_size":384,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3153920,"num_objects":49,"num_object_clones":0,"num_object_copies":147,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":385,"num_write_kb":3080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":2957312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":296,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-22T03:18:21.077512+0000","last_change":"2024-09-22T03:18:19.078674+0000","last_active":"2024-09-22T03:18:21.077512+0000","last_peered":"2024-09-22T03:18:21.077512+0000","last_clean":"2024-09-22T03:18:21.077512+0000","last_became_active":"2024-09-22T03:18:18.121868+0000","last_became_peered":"2024-09-22T03:18:18.121868+0000","last_unstale":"2024-09-22T03:18:21.077512+0000","last_undegraded":"2024-09-22T03:18:21.077512+0000","last_fullsized":"2024-09-22T03:18:21.077512+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.b","version":"123'296","reported_seq":344,"reported_epoch":124,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-22T03:18:21.908775+0000","last_change":"2024-09-22T03:18:21.839090+0000","last_active":"2024-09-22T03:18:21.908775+0000","last_peered":"2024-09-22T03:18:21.908775+0000","last_clean":"2024-09-22T03:17:28.441691+0000","last_became_active":"2024-09-22T03:18:19.322920+0000","last_became_peered":"2024-09-22T03:18:19.322920+0000","last_unstale":"2024-09-22T03:18:21.908775+0000","last_undegraded":"2024-09-22T03:18:19.116974+0000","last_fullsized":"2024-09-22T03:18:19.109309+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:17:25.017919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:17:25.017919+0000","last_clean_scrub_stamp":"2024-09-22T03:17:25.017919+0000","objects_scrubbed":0,"log_size":296,"log_dups_size":0,"ondisk_log_size":296,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2424832,"num_objects":37,"num_object_clones":0,"num_object_copies":111,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":296,"num_write_kb":2368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":393216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,2","objects":12}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":293,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-22T03:18:21.047269+0000","last_change":"2024-09-22T03:18:19.067090+0000","last_active":"2024-09-22T03:18:21.047269+0000","last_peered":"2024-09-22T03:18:21.047269+0000","last_clean":"2024-09-22T03:18:21.047269+0000","last_became_active":"2024-09-22T03:18:18.104717+0000","last_became_peered":"2024-09-22T03:18:18.104717+0000","last_unstale":"2024-09-22T03:18:21.047269+0000","last_undegraded":"2024-09-22T03:18:21.047269+0000","last_fullsized":"2024-09-22T03:18:21.047269+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.a","version":"123'311","reported_seq":322,"reported_epoch":123,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-22T03:18:20.074408+0000","last_change":"2024-09-22T03:18:19.400988+0000","last_active":"2024-09-22T03:18:20.074408+0000","last_peered":"2024-09-22T03:18:20.074408+0000","last_clean":"2024-09-22T03:17:25.017919+0000","last_became_active":"2024-09-22T03:18:19.395317+0000","last_became_peered":"2024-09-22T03:18:19.395317+0000","last_unstale":"2024-09-22T03:18:20.074408+0000","last_undegraded":"2024-09-22T03:18:19.062506+0000","last_fullsized":"2024-09-22T03:18:19.057039+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:17:25.017919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:17:25.017919+0000","last_clean_scrub_stamp":"2024-09-22T03:17:25.017919+0000","objects_scrubbed":0,"log_size":311,"log_dups_size":0,"ondisk_log_size":311,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2555904,"num_objects":39,"num_object_clones":0,"num_object_copies":117,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":312,"num_write_kb":2496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":183,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-09-22T03:18:19.046544+0000","last_change":"2024-09-22T03:18:19.050753+0000","last_active":"2024-09-22T03:18:19.046544+0000","last_peered":"2024-09-22T03:18:19.046544+0000","last_clean":"2024-09-22T03:18:19.046544+0000","last_became_active":"2024-09-22T03:18:18.099051+0000","last_became_peered":"2024-09-22T03:18:18.099051+0000","last_unstale":"2024-09-22T03:18:19.046544+0000","last_undegraded":"2024-09-22T03:18:19.046544+0000","last_fullsized":"2024-09-22T03:18:19.046544+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"12'97","reported_seq":417,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-22T03:18:21.080339+0000","last_change":"2024-09-22T03:18:18.126160+0000","last_active":"2024-09-22T03:18:21.080339+0000","last_peered":"2024-09-22T03:18:21.080339+0000","last_clean":"2024-09-22T03:18:21.080339+0000","last_became_active":"2024-09-22T03:18:18.122292+0000","last_became_peered":"2024-09-22T03:18:18.122292+0000","last_unstale":"2024-09-22T03:18:21.080339+0000","last_undegraded":"2024-09-22T03:18:21.080339+0000","last_fullsized":"2024-09-22T03:18:21.080339+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.d","version":"123'320","reported_seq":357,"reported_epoch":124,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-22T03:18:21.081962+0000","last_change":"2024-09-22T03:18:19.412042+0000","last_active":"2024-09-22T03:18:21.081962+0000","last_peered":"2024-09-22T03:18:21.081962+0000","last_clean":"2024-09-22T03:17:28.160849+0000","last_became_active":"2024-09-22T03:18:19.404230+0000","last_became_peered":"2024-09-22T03:18:19.404230+0000","last_unstale":"2024-09-22T03:18:21.081962+0000","last_undegraded":"2024-09-22T03:18:19.088117+0000","last_fullsized":"2024-09-22T03:18:19.081933+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:17:25.017919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:17:25.017919+0000","last_clean_scrub_stamp":"2024-09-22T03:17:25.017919+0000","objects_scrubbed":0,"log_size":320,"log_dups_size":0,"ondisk_log_size":320,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2621440,"num_objects":40,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":320,"num_write_kb":2560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,2","objects":5}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":249,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-22T03:18:22.060900+0000","last_change":"2024-09-22T03:18:19.044456+0000","last_active":"2024-09-22T03:18:22.060900+0000","last_peered":"2024-09-22T03:18:22.060900+0000","last_clean":"2024-09-22T03:18:22.060900+0000","last_became_active":"2024-09-22T03:18:18.129858+0000","last_became_peered":"2024-09-22T03:18:18.129858+0000","last_unstale":"2024-09-22T03:18:22.060900+0000","last_undegraded":"2024-09-22T03:18:22.060900+0000","last_fullsized":"2024-09-22T03:18:22.060900+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.c","version":"124'431","reported_seq":473,"reported_epoch":124,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-22T03:18:21.083127+0000","last_change":"2024-09-22T03:18:19.404282+0000","last_active":"2024-09-22T03:18:21.083127+0000","last_peered":"2024-09-22T03:18:21.083127+0000","last_clean":"2024-09-22T03:17:25.017919+0000","last_became_active":"2024-09-22T03:18:19.398884+0000","last_became_peered":"2024-09-22T03:18:19.398884+0000","last_unstale":"2024-09-22T03:18:21.083127+0000","last_undegraded":"2024-09-22T03:18:19.061071+0000","last_fullsized":"2024-09-22T03:18:19.052000+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:17:25.017919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:17:25.017919+0000","last_clean_scrub_stamp":"2024-09-22T03:17:25.017919+0000","objects_scrubbed":0,"log_size":431,"log_dups_size":0,"ondisk_log_size":431,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3538944,"num_objects":54,"num_object_clones":0,"num_object_copies":162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":49,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":432,"num_write_kb":3456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":5}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":268,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-22T03:18:21.047791+0000","last_change":"2024-09-22T03:18:17.081823+0000","last_active":"2024-09-22T03:18:21.047791+0000","last_peered":"2024-09-22T03:18:21.047791+0000","last_clean":"2024-09-22T03:18:21.047791+0000","last_became_active":"2024-09-22T03:18:01.232732+0000","last_became_peered":"2024-09-22T03:18:01.232732+0000","last_unstale":"2024-09-22T03:18:21.047791+0000","last_undegraded":"2024-09-22T03:18:21.047791+0000","last_fullsized":"2024-09-22T03:18:21.047791+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.f","version":"123'391","reported_seq":406,"reported_epoch":123,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-22T03:18:19.682193+0000","last_change":"2024-09-22T03:18:19.394293+0000","last_active":"2024-09-22T03:18:19.682193+0000","last_peered":"2024-09-22T03:18:19.682193+0000","last_clean":"2024-09-22T03:17:25.017919+0000","last_became_active":"2024-09-22T03:18:19.391348+0000","last_became_peered":"2024-09-22T03:18:19.391348+0000","last_unstale":"2024-09-22T03:18:19.682193+0000","last_undegraded":"2024-09-22T03:18:19.043600+0000","last_fullsized":"2024-09-22T03:18:19.036283+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:17:25.017919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:17:25.017919+0000","last_clean_scrub_stamp":"2024-09-22T03:17:25.017919+0000","objects_scrubbed":0,"log_size":391,"log_dups_size":0,"ondisk_log_size":391,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3211264,"num_objects":49,"num_object_clones":0,"num_object_copies":147,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":46,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":392,"num_write_kb":3136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":297,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-22T03:18:22.047685+0000","last_change":"2024-09-22T03:18:19.043166+0000","last_active":"2024-09-22T03:18:22.047685+0000","last_peered":"2024-09-22T03:18:22.047685+0000","last_clean":"2024-09-22T03:18:22.047685+0000","last_became_active":"2024-09-22T03:18:18.098084+0000","last_became_peered":"2024-09-22T03:18:18.098084+0000","last_unstale":"2024-09-22T03:18:22.047685+0000","last_undegraded":"2024-09-22T03:18:22.047685+0000","last_fullsized":"2024-09-22T03:18:22.047685+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.e","version":"123'352","reported_seq":393,"reported_epoch":124,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-22T03:18:21.079821+0000","last_change":"2024-09-22T03:18:19.436894+0000","last_active":"2024-09-22T03:18:21.079821+0000","last_peered":"2024-09-22T03:18:21.079821+0000","last_clean":"2024-09-22T03:17:25.017919+0000","last_became_active":"2024-09-22T03:18:19.428685+0000","last_became_peered":"2024-09-22T03:18:19.428685+0000","last_unstale":"2024-09-22T03:18:21.079821+0000","last_undegraded":"2024-09-22T03:18:19.093276+0000","last_fullsized":"2024-09-22T03:18:19.081651+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:17:25.017919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:17:25.017919+0000","last_clean_scrub_stamp":"2024-09-22T03:17:25.017919+0000","objects_scrubbed":0,"log_size":352,"log_dups_size":0,"ondisk_log_size":352,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2883584,"num_objects":44,"num_object_clones":0,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":40,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":352,"num_write_kb":2816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":4}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":290,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-22T03:18:22.050428+0000","last_change":"2024-09-22T03:18:19.042797+0000","last_active":"2024-09-22T03:18:22.050428+0000","last_peered":"2024-09-22T03:18:22.050428+0000","last_clean":"2024-09-22T03:18:22.050428+0000","last_became_active":"2024-09-22T03:18:18.118143+0000","last_became_peered":"2024-09-22T03:18:18.118143+0000","last_unstale":"2024-09-22T03:18:22.050428+0000","last_undegraded":"2024-09-22T03:18:22.050428+0000","last_fullsized":"2024-09-22T03:18:22.050428+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":51191808,"num_objects":782,"num_object_clones":0,"num_object_copies":2346,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":370,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6249,"num_write_kb":49992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":259,"num_bytes_recovered":15966208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":57761792,"data_stored":140394496,"data_compressed":7136536,"data_compressed_allocated":57761792,"data_compressed_original":140394496,"omap_allocated":0,"internal_metadata":0},"log_size":6243,"ondisk_log_size":6243,"up":48,"acting":36,"num_store_stats":4},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1180736,"data_compressed":59924,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":119,"seq":511101108244,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":49328,"kb_used_data":18400,"kb_used_omap":0,"kb_used_meta":30912,"kb_avail":94322512,"statfs":{"total":96636764160,"available":96586252288,"internally_reserved":0,"allocated":18841600,"data_stored":38433911,"data_compressed":1951998,"data_compressed_allocated":17805312,"data_compressed_original":38019072,"omap_allocated":0,"internal_metadata":31653888},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":100,"seq":429496729724,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":55904,"kb_used_data":14992,"kb_used_omap":0,"kb_used_meta":40896,"kb_avail":94315936,"statfs":{"total":96636764160,"available":96579518464,"internally_reserved":0,"allocated":15351808,"data_stored":29513367,"data_compressed":1501350,"data_compressed_allocated":14295040,"data_compressed_original":29081600,"omap_allocated":0,"internal_metadata":41877504},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":121,"seq":519691042821,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":43040,"kb_used_data":5520,"kb_used_omap":0,"kb_used_meta":37504,"kb_avail":94328800,"statfs":{"total":96636764160,"available":96592691200,"internally_reserved":0,"allocated":5652480,"data_stored":32011011,"data_compressed":1532892,"data_compressed_allocated":4579328,"data_compressed_original":31571968,"omap_allocated":0,"internal_metadata":38404096},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":73,"seq":313532612833,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":75576,"kb_used_data":21224,"kb_used_omap":0,"kb_used_meta":54336,"kb_avail":94296264,"statfs":{"total":96636764160,"available":96559374336,"internally_reserved":0,"allocated":21733376,"data_stored":42296531,"data_compressed":2160829,"data_compressed_allocated":20672512,"data_compressed_original":41861120,"omap_allocated":0,"internal_metadata":55640064},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29962,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29962,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20631552,"data_stored":41287680,"data_compressed":2130867,"data_compressed_allocated":20631552,"data_compressed_original":41287680,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4612096,"data_stored":31662080,"data_compressed":1534907,"data_compressed_allocated":4612096,"data_compressed_original":31662080,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":14385152,"data_stored":28770304,"data_compressed":1484924,"data_compressed_allocated":14385152,"data_compressed_original":28770304,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":18132992,"data_stored":38674432,"data_compressed":1985838,"data_compressed_allocated":18132992,"data_compressed_original":38674432,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-22T03:18:25.787 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-22T03:18:25.929 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:18:25.923+0000 7f1fe7e6c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:18:25.936 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:18:25.934+0000 7f1fe7e6c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:18:26.112 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 59 16 6742 6726 0.890523 2.25781 0.0389285 0.140258 2024-09-22T03:18:26.279 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-22T03:18:26.279 INFO:teuthology.orchestra.run.smithi153.stdout:{"fsid":"f17abef2-bc08-46d2-abf1-d47a95f0e2ee","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 370/2358 objects degraded (15.691%), 10 pgs degraded","count":10},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":632,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":128,"num_osds":4,"num_up_osds":4,"osd_up_since":1726975097,"num_in_osds":4,"osd_in_since":1726974475,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":13},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":7},{"state_name":"active+recovering+undersized+degraded+remapped","count":3},{"state_name":"active+undersized+remapped","count":2}],"num_pgs":25,"num_pools":3,"num_objects":786,"data_bytes":51782195,"bytes_used":229220352,"bytes_avail":386317836288,"bytes_total":386547056640,"degraded_objects":370,"degraded_total":2358,"degraded_ratio":0.15691263782866835,"recovering_objects_per_sec":54,"recovering_bytes_per_sec":3532024,"recovering_keys_per_sec":0,"num_objects_recovered":217,"num_bytes_recovered":14131200,"num_keys_recovered":0,"write_bytes_sec":1203959,"read_op_per_sec":0,"write_op_per_sec":146},"fsmap":{"epoch":1,"btime":"2024-09-22T03:07:53:859133+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":2,"modified":"2024-09-22T03:09:00.114736+0000","services":{}},"progress_events":{}} 2024-09-22T03:18:26.297 INFO:tasks.ceph.ceph_manager.ceph:making progress, resetting timeout 2024-09-22T03:18:26.298 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-22T03:18:26.429 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:18:26.423+0000 7f0f67b97640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:18:26.432 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:18:26.431+0000 7f0f67b97640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:18:26.654 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-22T03:18:26.655 INFO:teuthology.orchestra.run.smithi153.stderr:dumped all 2024-09-22T03:18:26.671 INFO:teuthology.orchestra.run.smithi153.stdout:{"pg_ready":true,"pg_map":{"version":433,"stamp":"2024-09-22T03:18:26.240171+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":52437555,"num_objects":796,"num_object_clones":0,"num_object_copies":2388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":237,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":6468,"num_write_kb":52528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":392,"num_bytes_recovered":24666112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6425,"ondisk_log_size":6425,"up":75,"acting":66,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":225260,"kb_used_data":61548,"kb_used_omap":0,"kb_used_meta":163648,"kb_avail":377262100,"statfs":{"total":386547056640,"available":386316390400,"internally_reserved":0,"allocated":63025152,"data_stored":145082098,"data_compressed":7292158,"data_compressed_allocated":58757120,"data_compressed_original":143343616,"omap_allocated":0,"internal_metadata":167575552},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1253376,"num_objects":20,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":153,"num_write_kb":1224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":214,"num_bytes_recovered":13950976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.202640"},"pg_stats":[{"pgid":"8.7","version":"124'504","reported_seq":630,"reported_epoch":124,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-22T03:18:21.674252+0000","last_change":"2024-09-22T03:18:21.164281+0000","last_active":"2024-09-22T03:18:21.674252+0000","last_peered":"2024-09-22T03:18:21.674252+0000","last_clean":"2024-09-22T03:17:25.017919+0000","last_became_active":"2024-09-22T03:18:19.397637+0000","last_became_peered":"2024-09-22T03:18:19.397637+0000","last_unstale":"2024-09-22T03:18:21.674252+0000","last_undegraded":"2024-09-22T03:18:19.061412+0000","last_fullsized":"2024-09-22T03:18:19.053844+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:17:25.017919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:17:25.017919+0000","last_clean_scrub_stamp":"2024-09-22T03:17:25.017919+0000","objects_scrubbed":0,"log_size":504,"log_dups_size":0,"ondisk_log_size":504,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4128768,"num_objects":63,"num_object_clones":0,"num_object_copies":189,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":504,"num_write_kb":4032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":2883584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":48}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.6","version":"124'352","reported_seq":460,"reported_epoch":124,"state":"active+undersized+remapped","last_fresh":"2024-09-22T03:18:21.397439+0000","last_change":"2024-09-22T03:18:21.398718+0000","last_active":"2024-09-22T03:18:21.397439+0000","last_peered":"2024-09-22T03:18:21.397439+0000","last_clean":"2024-09-22T03:17:28.360944+0000","last_became_active":"2024-09-22T03:18:19.318455+0000","last_became_peered":"2024-09-22T03:18:19.318455+0000","last_unstale":"2024-09-22T03:18:21.397439+0000","last_undegraded":"2024-09-22T03:18:19.051817+0000","last_fullsized":"2024-09-22T03:18:19.036657+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:17:25.017919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:17:25.017919+0000","last_clean_scrub_stamp":"2024-09-22T03:17:25.017919+0000","objects_scrubbed":0,"log_size":352,"log_dups_size":0,"ondisk_log_size":352,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2883584,"num_objects":44,"num_object_clones":0,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":352,"num_write_kb":2816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":2260992,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0],"avail_no_missing":["2","0","1"],"object_location_counts":[{"shards":"0,2","objects":44}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.5","version":"123'432","reported_seq":481,"reported_epoch":124,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-22T03:18:21.913645+0000","last_change":"2024-09-22T03:18:21.832176+0000","last_active":"2024-09-22T03:18:21.913645+0000","last_peered":"2024-09-22T03:18:21.913645+0000","last_clean":"2024-09-22T03:17:28.351270+0000","last_became_active":"2024-09-22T03:18:19.319765+0000","last_became_peered":"2024-09-22T03:18:19.319765+0000","last_unstale":"2024-09-22T03:18:21.913645+0000","last_undegraded":"2024-09-22T03:18:19.107317+0000","last_fullsized":"2024-09-22T03:18:19.094542+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:17:25.017919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:17:25.017919+0000","last_clean_scrub_stamp":"2024-09-22T03:17:25.017919+0000","objects_scrubbed":0,"log_size":432,"log_dups_size":0,"ondisk_log_size":432,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3538944,"num_objects":54,"num_object_clones":0,"num_object_copies":162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":37,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":432,"num_write_kb":3456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":458752,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,2","objects":17}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.4","version":"123'464","reported_seq":605,"reported_epoch":124,"state":"active+undersized+remapped","last_fresh":"2024-09-22T03:18:21.118315+0000","last_change":"2024-09-22T03:18:21.120223+0000","last_active":"2024-09-22T03:18:21.118315+0000","last_peered":"2024-09-22T03:18:21.118315+0000","last_clean":"2024-09-22T03:17:25.017919+0000","last_became_active":"2024-09-22T03:18:19.253832+0000","last_became_peered":"2024-09-22T03:18:19.253832+0000","last_unstale":"2024-09-22T03:18:21.118315+0000","last_undegraded":"2024-09-22T03:18:19.080228+0000","last_fullsized":"2024-09-22T03:18:19.066463+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:17:25.017919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:17:25.017919+0000","last_clean_scrub_stamp":"2024-09-22T03:17:25.017919+0000","objects_scrubbed":0,"log_size":464,"log_dups_size":0,"ondisk_log_size":464,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3801088,"num_objects":58,"num_object_clones":0,"num_object_copies":174,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":464,"num_write_kb":3712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":51,"num_bytes_recovered":3342336,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,3],"avail_no_missing":["0","1","3"],"object_location_counts":[{"shards":"0,3","objects":58}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.3","version":"123'328","reported_seq":325,"reported_epoch":124,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-22T03:18:21.080958+0000","last_change":"2024-09-22T03:18:19.414081+0000","last_active":"2024-09-22T03:18:21.080958+0000","last_peered":"2024-09-22T03:18:21.080958+0000","last_clean":"2024-09-22T03:17:25.017919+0000","last_became_active":"2024-09-22T03:18:19.409234+0000","last_became_peered":"2024-09-22T03:18:19.409234+0000","last_unstale":"2024-09-22T03:18:21.080958+0000","last_undegraded":"2024-09-22T03:18:19.072687+0000","last_fullsized":"2024-09-22T03:18:19.066510+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:17:25.017919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:17:25.017919+0000","last_clean_scrub_stamp":"2024-09-22T03:17:25.017919+0000","objects_scrubbed":0,"log_size":328,"log_dups_size":0,"ondisk_log_size":328,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2686976,"num_objects":41,"num_object_clones":0,"num_object_copies":123,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":328,"num_write_kb":2624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":5}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.2","version":"123'432","reported_seq":483,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-22T03:18:21.081501+0000","last_change":"2024-09-22T03:18:01.847644+0000","last_active":"2024-09-22T03:18:21.081501+0000","last_peered":"2024-09-22T03:18:21.081501+0000","last_clean":"2024-09-22T03:18:21.081501+0000","last_became_active":"2024-09-22T03:18:01.225010+0000","last_became_peered":"2024-09-22T03:18:01.225010+0000","last_unstale":"2024-09-22T03:18:21.081501+0000","last_undegraded":"2024-09-22T03:18:21.081501+0000","last_fullsized":"2024-09-22T03:18:21.081501+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:17:25.017919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:17:25.017919+0000","last_clean_scrub_stamp":"2024-09-22T03:17:25.017919+0000","objects_scrubbed":0,"log_size":432,"log_dups_size":0,"ondisk_log_size":432,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3538944,"num_objects":54,"num_object_clones":0,"num_object_copies":162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":432,"num_write_kb":3456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":393216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1","version":"127'480","reported_seq":590,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-22T03:18:24.939913+0000","last_change":"2024-09-22T03:18:23.981632+0000","last_active":"2024-09-22T03:18:24.939913+0000","last_peered":"2024-09-22T03:18:24.939913+0000","last_clean":"2024-09-22T03:18:24.939913+0000","last_became_active":"2024-09-22T03:18:23.978972+0000","last_became_peered":"2024-09-22T03:18:23.978972+0000","last_unstale":"2024-09-22T03:18:24.939913+0000","last_undegraded":"2024-09-22T03:18:24.939913+0000","last_fullsized":"2024-09-22T03:18:24.939913+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:17:25.017919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:17:25.017919+0000","last_clean_scrub_stamp":"2024-09-22T03:17:25.017919+0000","objects_scrubbed":0,"log_size":480,"log_dups_size":0,"ondisk_log_size":480,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3932160,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":480,"num_write_kb":3840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":3342336,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"123'344","reported_seq":429,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-22T03:18:22.111803+0000","last_change":"2024-09-22T03:18:22.112809+0000","last_active":"2024-09-22T03:18:22.111803+0000","last_peered":"2024-09-22T03:18:22.111803+0000","last_clean":"2024-09-22T03:17:25.017919+0000","last_became_active":"2024-09-22T03:18:22.107439+0000","last_became_peered":"2024-09-22T03:18:22.107439+0000","last_unstale":"2024-09-22T03:18:22.111803+0000","last_undegraded":"2024-09-22T03:18:22.111803+0000","last_fullsized":"2024-09-22T03:18:22.111803+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:17:25.017919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:17:25.017919+0000","last_clean_scrub_stamp":"2024-09-22T03:17:25.017919+0000","objects_scrubbed":0,"log_size":344,"log_dups_size":0,"ondisk_log_size":344,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2818048,"num_objects":43,"num_object_clones":0,"num_object_copies":129,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":344,"num_write_kb":2752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":2490368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.9","version":"124'439","reported_seq":488,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-22T03:18:21.170976+0000","last_change":"2024-09-22T03:18:01.629983+0000","last_active":"2024-09-22T03:18:21.170976+0000","last_peered":"2024-09-22T03:18:21.170976+0000","last_clean":"2024-09-22T03:18:21.170976+0000","last_became_active":"2024-09-22T03:18:01.196590+0000","last_became_peered":"2024-09-22T03:18:01.196590+0000","last_unstale":"2024-09-22T03:18:21.170976+0000","last_undegraded":"2024-09-22T03:18:21.170976+0000","last_fullsized":"2024-09-22T03:18:21.170976+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:17:25.017919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:17:25.017919+0000","last_clean_scrub_stamp":"2024-09-22T03:17:25.017919+0000","objects_scrubbed":0,"log_size":439,"log_dups_size":0,"ondisk_log_size":439,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3604480,"num_objects":55,"num_object_clones":0,"num_object_copies":165,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":440,"num_write_kb":3520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":393216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":176,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-22T03:18:24.940295+0000","last_change":"2024-09-22T03:18:17.050118+0000","last_active":"2024-09-22T03:18:24.940295+0000","last_peered":"2024-09-22T03:18:24.940295+0000","last_clean":"2024-09-22T03:18:24.940295+0000","last_became_active":"2024-09-22T03:18:01.415712+0000","last_became_peered":"2024-09-22T03:18:01.415712+0000","last_unstale":"2024-09-22T03:18:24.940295+0000","last_undegraded":"2024-09-22T03:18:24.940295+0000","last_fullsized":"2024-09-22T03:18:24.940295+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.8","version":"125'384","reported_seq":524,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-22T03:18:22.143102+0000","last_change":"2024-09-22T03:18:22.093165+0000","last_active":"2024-09-22T03:18:22.143102+0000","last_peered":"2024-09-22T03:18:22.143102+0000","last_clean":"2024-09-22T03:18:22.143102+0000","last_became_active":"2024-09-22T03:18:22.084001+0000","last_became_peered":"2024-09-22T03:18:22.084001+0000","last_unstale":"2024-09-22T03:18:22.143102+0000","last_undegraded":"2024-09-22T03:18:22.143102+0000","last_fullsized":"2024-09-22T03:18:22.143102+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:17:25.017919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:17:25.017919+0000","last_clean_scrub_stamp":"2024-09-22T03:17:25.017919+0000","objects_scrubbed":0,"log_size":384,"log_dups_size":0,"ondisk_log_size":384,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3153920,"num_objects":49,"num_object_clones":0,"num_object_copies":147,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":385,"num_write_kb":3080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":2957312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":296,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-22T03:18:21.077512+0000","last_change":"2024-09-22T03:18:19.078674+0000","last_active":"2024-09-22T03:18:21.077512+0000","last_peered":"2024-09-22T03:18:21.077512+0000","last_clean":"2024-09-22T03:18:21.077512+0000","last_became_active":"2024-09-22T03:18:18.121868+0000","last_became_peered":"2024-09-22T03:18:18.121868+0000","last_unstale":"2024-09-22T03:18:21.077512+0000","last_undegraded":"2024-09-22T03:18:21.077512+0000","last_fullsized":"2024-09-22T03:18:21.077512+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.b","version":"123'296","reported_seq":344,"reported_epoch":124,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-22T03:18:21.908775+0000","last_change":"2024-09-22T03:18:21.839090+0000","last_active":"2024-09-22T03:18:21.908775+0000","last_peered":"2024-09-22T03:18:21.908775+0000","last_clean":"2024-09-22T03:17:28.441691+0000","last_became_active":"2024-09-22T03:18:19.322920+0000","last_became_peered":"2024-09-22T03:18:19.322920+0000","last_unstale":"2024-09-22T03:18:21.908775+0000","last_undegraded":"2024-09-22T03:18:19.116974+0000","last_fullsized":"2024-09-22T03:18:19.109309+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:17:25.017919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:17:25.017919+0000","last_clean_scrub_stamp":"2024-09-22T03:17:25.017919+0000","objects_scrubbed":0,"log_size":296,"log_dups_size":0,"ondisk_log_size":296,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2424832,"num_objects":37,"num_object_clones":0,"num_object_copies":111,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":296,"num_write_kb":2368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":393216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,2","objects":12}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":293,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-22T03:18:21.047269+0000","last_change":"2024-09-22T03:18:19.067090+0000","last_active":"2024-09-22T03:18:21.047269+0000","last_peered":"2024-09-22T03:18:21.047269+0000","last_clean":"2024-09-22T03:18:21.047269+0000","last_became_active":"2024-09-22T03:18:18.104717+0000","last_became_peered":"2024-09-22T03:18:18.104717+0000","last_unstale":"2024-09-22T03:18:21.047269+0000","last_undegraded":"2024-09-22T03:18:21.047269+0000","last_fullsized":"2024-09-22T03:18:21.047269+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.a","version":"127'336","reported_seq":437,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-22T03:18:24.941405+0000","last_change":"2024-09-22T03:18:23.099898+0000","last_active":"2024-09-22T03:18:24.941405+0000","last_peered":"2024-09-22T03:18:24.941405+0000","last_clean":"2024-09-22T03:18:24.941405+0000","last_became_active":"2024-09-22T03:18:23.094343+0000","last_became_peered":"2024-09-22T03:18:23.094343+0000","last_unstale":"2024-09-22T03:18:24.941405+0000","last_undegraded":"2024-09-22T03:18:24.941405+0000","last_fullsized":"2024-09-22T03:18:24.941405+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:17:25.017919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:17:25.017919+0000","last_clean_scrub_stamp":"2024-09-22T03:17:25.017919+0000","objects_scrubbed":0,"log_size":336,"log_dups_size":0,"ondisk_log_size":336,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2752512,"num_objects":42,"num_object_clones":0,"num_object_copies":126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":336,"num_write_kb":2688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39,"num_bytes_recovered":2408448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":198,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-22T03:18:24.938427+0000","last_change":"2024-09-22T03:18:19.050753+0000","last_active":"2024-09-22T03:18:24.938427+0000","last_peered":"2024-09-22T03:18:24.938427+0000","last_clean":"2024-09-22T03:18:24.938427+0000","last_became_active":"2024-09-22T03:18:18.099051+0000","last_became_peered":"2024-09-22T03:18:18.099051+0000","last_unstale":"2024-09-22T03:18:24.938427+0000","last_undegraded":"2024-09-22T03:18:24.938427+0000","last_fullsized":"2024-09-22T03:18:24.938427+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"12'97","reported_seq":417,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-22T03:18:21.080339+0000","last_change":"2024-09-22T03:18:18.126160+0000","last_active":"2024-09-22T03:18:21.080339+0000","last_peered":"2024-09-22T03:18:21.080339+0000","last_clean":"2024-09-22T03:18:21.080339+0000","last_became_active":"2024-09-22T03:18:18.122292+0000","last_became_peered":"2024-09-22T03:18:18.122292+0000","last_unstale":"2024-09-22T03:18:21.080339+0000","last_undegraded":"2024-09-22T03:18:21.080339+0000","last_fullsized":"2024-09-22T03:18:21.080339+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.d","version":"123'320","reported_seq":357,"reported_epoch":124,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-22T03:18:21.081962+0000","last_change":"2024-09-22T03:18:19.412042+0000","last_active":"2024-09-22T03:18:21.081962+0000","last_peered":"2024-09-22T03:18:21.081962+0000","last_clean":"2024-09-22T03:17:28.160849+0000","last_became_active":"2024-09-22T03:18:19.404230+0000","last_became_peered":"2024-09-22T03:18:19.404230+0000","last_unstale":"2024-09-22T03:18:21.081962+0000","last_undegraded":"2024-09-22T03:18:19.088117+0000","last_fullsized":"2024-09-22T03:18:19.081933+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:17:25.017919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:17:25.017919+0000","last_clean_scrub_stamp":"2024-09-22T03:17:25.017919+0000","objects_scrubbed":0,"log_size":320,"log_dups_size":0,"ondisk_log_size":320,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2621440,"num_objects":40,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":320,"num_write_kb":2560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,2","objects":5}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":249,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-22T03:18:22.060900+0000","last_change":"2024-09-22T03:18:19.044456+0000","last_active":"2024-09-22T03:18:22.060900+0000","last_peered":"2024-09-22T03:18:22.060900+0000","last_clean":"2024-09-22T03:18:22.060900+0000","last_became_active":"2024-09-22T03:18:18.129858+0000","last_became_peered":"2024-09-22T03:18:18.129858+0000","last_unstale":"2024-09-22T03:18:22.060900+0000","last_undegraded":"2024-09-22T03:18:22.060900+0000","last_fullsized":"2024-09-22T03:18:22.060900+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.c","version":"124'431","reported_seq":473,"reported_epoch":124,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-22T03:18:21.083127+0000","last_change":"2024-09-22T03:18:19.404282+0000","last_active":"2024-09-22T03:18:21.083127+0000","last_peered":"2024-09-22T03:18:21.083127+0000","last_clean":"2024-09-22T03:17:25.017919+0000","last_became_active":"2024-09-22T03:18:19.398884+0000","last_became_peered":"2024-09-22T03:18:19.398884+0000","last_unstale":"2024-09-22T03:18:21.083127+0000","last_undegraded":"2024-09-22T03:18:19.061071+0000","last_fullsized":"2024-09-22T03:18:19.052000+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:17:25.017919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:17:25.017919+0000","last_clean_scrub_stamp":"2024-09-22T03:17:25.017919+0000","objects_scrubbed":0,"log_size":431,"log_dups_size":0,"ondisk_log_size":431,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3538944,"num_objects":54,"num_object_clones":0,"num_object_copies":162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":49,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":432,"num_write_kb":3456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":5}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":268,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-22T03:18:21.047791+0000","last_change":"2024-09-22T03:18:17.081823+0000","last_active":"2024-09-22T03:18:21.047791+0000","last_peered":"2024-09-22T03:18:21.047791+0000","last_clean":"2024-09-22T03:18:21.047791+0000","last_became_active":"2024-09-22T03:18:01.232732+0000","last_became_peered":"2024-09-22T03:18:01.232732+0000","last_unstale":"2024-09-22T03:18:21.047791+0000","last_undegraded":"2024-09-22T03:18:21.047791+0000","last_fullsized":"2024-09-22T03:18:21.047791+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.f","version":"128'431","reported_seq":557,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-22T03:18:25.305301+0000","last_change":"2024-09-22T03:18:23.087718+0000","last_active":"2024-09-22T03:18:25.305301+0000","last_peered":"2024-09-22T03:18:25.305301+0000","last_clean":"2024-09-22T03:18:25.305301+0000","last_became_active":"2024-09-22T03:18:23.080292+0000","last_became_peered":"2024-09-22T03:18:23.080292+0000","last_unstale":"2024-09-22T03:18:25.305301+0000","last_undegraded":"2024-09-22T03:18:25.305301+0000","last_fullsized":"2024-09-22T03:18:25.305301+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:17:25.017919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:17:25.017919+0000","last_clean_scrub_stamp":"2024-09-22T03:17:25.017919+0000","objects_scrubbed":0,"log_size":431,"log_dups_size":0,"ondisk_log_size":431,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3538944,"num_objects":54,"num_object_clones":0,"num_object_copies":162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":432,"num_write_kb":3456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":3014656,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":297,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-22T03:18:22.047685+0000","last_change":"2024-09-22T03:18:19.043166+0000","last_active":"2024-09-22T03:18:22.047685+0000","last_peered":"2024-09-22T03:18:22.047685+0000","last_clean":"2024-09-22T03:18:22.047685+0000","last_became_active":"2024-09-22T03:18:18.098084+0000","last_became_peered":"2024-09-22T03:18:18.098084+0000","last_unstale":"2024-09-22T03:18:22.047685+0000","last_undegraded":"2024-09-22T03:18:22.047685+0000","last_fullsized":"2024-09-22T03:18:22.047685+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.e","version":"123'352","reported_seq":393,"reported_epoch":124,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-22T03:18:21.079821+0000","last_change":"2024-09-22T03:18:19.436894+0000","last_active":"2024-09-22T03:18:21.079821+0000","last_peered":"2024-09-22T03:18:21.079821+0000","last_clean":"2024-09-22T03:17:25.017919+0000","last_became_active":"2024-09-22T03:18:19.428685+0000","last_became_peered":"2024-09-22T03:18:19.428685+0000","last_unstale":"2024-09-22T03:18:21.079821+0000","last_undegraded":"2024-09-22T03:18:19.093276+0000","last_fullsized":"2024-09-22T03:18:19.081651+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:17:25.017919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:17:25.017919+0000","last_clean_scrub_stamp":"2024-09-22T03:17:25.017919+0000","objects_scrubbed":0,"log_size":352,"log_dups_size":0,"ondisk_log_size":352,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2883584,"num_objects":44,"num_object_clones":0,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":40,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":352,"num_write_kb":2816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":4}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":290,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-22T03:18:22.050428+0000","last_change":"2024-09-22T03:18:19.042797+0000","last_active":"2024-09-22T03:18:22.050428+0000","last_peered":"2024-09-22T03:18:22.050428+0000","last_clean":"2024-09-22T03:18:22.050428+0000","last_became_active":"2024-09-22T03:18:18.118143+0000","last_became_peered":"2024-09-22T03:18:18.118143+0000","last_unstale":"2024-09-22T03:18:22.050428+0000","last_undegraded":"2024-09-22T03:18:22.050428+0000","last_fullsized":"2024-09-22T03:18:22.050428+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":51847168,"num_objects":792,"num_object_clones":0,"num_object_copies":2376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":237,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6329,"num_write_kb":50632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":392,"num_bytes_recovered":24666112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":59101184,"data_stored":143073280,"data_compressed":7274857,"data_compressed_allocated":59101184,"data_compressed_original":143073280,"omap_allocated":0,"internal_metadata":0},"log_size":6325,"ondisk_log_size":6325,"up":48,"acting":39,"num_store_stats":4},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1180736,"data_compressed":59924,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":119,"seq":511101108249,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":50740,"kb_used_data":19812,"kb_used_omap":0,"kb_used_meta":30912,"kb_avail":94321100,"statfs":{"total":96636764160,"available":96584806400,"internally_reserved":0,"allocated":20287488,"data_stored":41261189,"data_compressed":2097087,"data_compressed_allocated":19210240,"data_compressed_original":40828928,"omap_allocated":0,"internal_metadata":31653888},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":100,"seq":429496729724,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":55904,"kb_used_data":14992,"kb_used_omap":0,"kb_used_meta":40896,"kb_avail":94315936,"statfs":{"total":96636764160,"available":96579518464,"internally_reserved":0,"allocated":15351808,"data_stored":29513367,"data_compressed":1501350,"data_compressed_allocated":14295040,"data_compressed_original":29081600,"omap_allocated":0,"internal_metadata":41877504},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":121,"seq":519691042821,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":43040,"kb_used_data":5520,"kb_used_omap":0,"kb_used_meta":37504,"kb_avail":94328800,"statfs":{"total":96636764160,"available":96592691200,"internally_reserved":0,"allocated":5652480,"data_stored":32011011,"data_compressed":1532892,"data_compressed_allocated":4579328,"data_compressed_original":31571968,"omap_allocated":0,"internal_metadata":38404096},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":73,"seq":313532612833,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":75576,"kb_used_data":21224,"kb_used_omap":0,"kb_used_meta":54336,"kb_avail":94296264,"statfs":{"total":96636764160,"available":96559374336,"internally_reserved":0,"allocated":21733376,"data_stored":42296531,"data_compressed":2160829,"data_compressed_allocated":20672512,"data_compressed_original":41861120,"omap_allocated":0,"internal_metadata":55640064},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29962,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29962,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20631552,"data_stored":41287680,"data_compressed":2130867,"data_compressed_allocated":20631552,"data_compressed_original":41287680,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4612096,"data_stored":31662080,"data_compressed":1534907,"data_compressed_allocated":4612096,"data_compressed_original":31662080,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":14385152,"data_stored":28770304,"data_compressed":1484924,"data_compressed_allocated":14385152,"data_compressed_original":28770304,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":19472384,"data_stored":41353216,"data_compressed":2124159,"data_compressed_allocated":19472384,"data_compressed_original":41353216,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-22T03:18:27.112 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:2024-09-22T03:18:27.111225+0000 min lat: 0.00665283 max lat: 33.1634 avg lat: 0.136841 2024-09-22T03:18:27.112 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-22T03:18:27.112 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 60 16 7028 7012 0.912917 2.23438 0.0255672 0.136841 2024-09-22T03:18:27.275 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-22T03:18:27.404 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:18:27.401+0000 7fba33258640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:18:27.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:18:27.406+0000 7fba33258640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:18:28.112 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 61 16 7348 7332 0.938931 2.5 0.0433808 0.133046 2024-09-22T03:18:29.112 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 62 16 7639 7623 0.960451 2.27344 0.0333034 0.130062 2024-09-22T03:18:29.596 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-22T03:18:29.674 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-22T03:18:29.715 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:18:29.712+0000 7f8492d17640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:18:29.720 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:18:29.718+0000 7f8492d17640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:18:29.810 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:18:29.804+0000 7f2567157640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:18:29.818 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:18:29.816+0000 7f2567157640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:18:30.051 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-22T03:18:30.051 INFO:teuthology.orchestra.run.smithi153.stderr:dumped all 2024-09-22T03:18:30.067 INFO:teuthology.orchestra.run.smithi153.stdout:{"pg_ready":true,"pg_map":{"version":434,"stamp":"2024-09-22T03:18:28.240622+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":56836659,"num_objects":865,"num_object_clones":0,"num_object_copies":2595,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":7005,"num_write_kb":56824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":629,"num_bytes_recovered":40198144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6953,"ondisk_log_size":6953,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":235648,"kb_used_data":67776,"kb_used_omap":0,"kb_used_meta":167808,"kb_avail":377251712,"statfs":{"total":386547056640,"available":386305753088,"internally_reserved":0,"allocated":69402624,"data_stored":169322593,"data_compressed":8490049,"data_compressed_allocated":65036288,"data_compressed_original":167542784,"omap_allocated":0,"internal_metadata":171835392},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":5054464,"num_objects":79,"num_object_clones":0,"num_object_copies":237,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-370,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":617,"num_write_kb":4936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":370,"num_bytes_recovered":24231936,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001338"},"pg_stats":[{"pgid":"8.7","version":"128'530","reported_seq":703,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-22T03:18:26.675057+0000","last_change":"2024-09-22T03:18:23.084812+0000","last_active":"2024-09-22T03:18:26.675057+0000","last_peered":"2024-09-22T03:18:26.675057+0000","last_clean":"2024-09-22T03:18:26.675057+0000","last_became_active":"2024-09-22T03:18:23.081537+0000","last_became_peered":"2024-09-22T03:18:23.081537+0000","last_unstale":"2024-09-22T03:18:26.675057+0000","last_undegraded":"2024-09-22T03:18:26.675057+0000","last_fullsized":"2024-09-22T03:18:26.675057+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:17:25.017919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:17:25.017919+0000","last_clean_scrub_stamp":"2024-09-22T03:17:25.017919+0000","objects_scrubbed":0,"log_size":530,"log_dups_size":0,"ondisk_log_size":530,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4349952,"num_objects":67,"num_object_clones":0,"num_object_copies":201,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":531,"num_write_kb":4248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":3866624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.6","version":"128'383","reported_seq":507,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-22T03:18:26.080594+0000","last_change":"2024-09-22T03:18:23.108631+0000","last_active":"2024-09-22T03:18:26.080594+0000","last_peered":"2024-09-22T03:18:26.080594+0000","last_clean":"2024-09-22T03:18:26.080594+0000","last_became_active":"2024-09-22T03:18:23.103912+0000","last_became_peered":"2024-09-22T03:18:23.103912+0000","last_unstale":"2024-09-22T03:18:26.080594+0000","last_undegraded":"2024-09-22T03:18:26.080594+0000","last_fullsized":"2024-09-22T03:18:26.080594+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:17:25.017919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:17:25.017919+0000","last_clean_scrub_stamp":"2024-09-22T03:17:25.017919+0000","objects_scrubbed":0,"log_size":383,"log_dups_size":0,"ondisk_log_size":383,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3145728,"num_objects":48,"num_object_clones":0,"num_object_copies":144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":384,"num_write_kb":3072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":2260992,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.5","version":"128'523","reported_seq":570,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-22T03:18:27.143049+0000","last_change":"2024-09-22T03:18:23.988488+0000","last_active":"2024-09-22T03:18:27.143049+0000","last_peered":"2024-09-22T03:18:27.143049+0000","last_clean":"2024-09-22T03:18:27.143049+0000","last_became_active":"2024-09-22T03:18:23.976152+0000","last_became_peered":"2024-09-22T03:18:23.976152+0000","last_unstale":"2024-09-22T03:18:27.143049+0000","last_undegraded":"2024-09-22T03:18:27.143049+0000","last_fullsized":"2024-09-22T03:18:27.143049+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:17:25.017919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:17:25.017919+0000","last_clean_scrub_stamp":"2024-09-22T03:17:25.017919+0000","objects_scrubbed":0,"log_size":523,"log_dups_size":0,"ondisk_log_size":523,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4292608,"num_objects":66,"num_object_clones":0,"num_object_copies":198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":524,"num_write_kb":4192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":2883584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.4","version":"128'487","reported_seq":644,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-22T03:18:26.787803+0000","last_change":"2024-09-22T03:18:23.121447+0000","last_active":"2024-09-22T03:18:26.787803+0000","last_peered":"2024-09-22T03:18:26.787803+0000","last_clean":"2024-09-22T03:18:26.787803+0000","last_became_active":"2024-09-22T03:18:23.106783+0000","last_became_peered":"2024-09-22T03:18:23.106783+0000","last_unstale":"2024-09-22T03:18:26.787803+0000","last_undegraded":"2024-09-22T03:18:26.787803+0000","last_fullsized":"2024-09-22T03:18:26.787803+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:17:25.017919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:17:25.017919+0000","last_clean_scrub_stamp":"2024-09-22T03:17:25.017919+0000","objects_scrubbed":0,"log_size":487,"log_dups_size":0,"ondisk_log_size":487,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3997696,"num_objects":61,"num_object_clones":0,"num_object_copies":183,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":488,"num_write_kb":3904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":51,"num_bytes_recovered":3342336,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.3","version":"128'335","reported_seq":342,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-22T03:18:26.460544+0000","last_change":"2024-09-22T03:18:24.983407+0000","last_active":"2024-09-22T03:18:26.460544+0000","last_peered":"2024-09-22T03:18:26.460544+0000","last_clean":"2024-09-22T03:18:26.460544+0000","last_became_active":"2024-09-22T03:18:24.979743+0000","last_became_peered":"2024-09-22T03:18:24.979743+0000","last_unstale":"2024-09-22T03:18:26.460544+0000","last_undegraded":"2024-09-22T03:18:26.460544+0000","last_fullsized":"2024-09-22T03:18:26.460544+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:17:25.017919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:17:25.017919+0000","last_clean_scrub_stamp":"2024-09-22T03:17:25.017919+0000","objects_scrubbed":0,"log_size":335,"log_dups_size":0,"ondisk_log_size":335,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2752512,"num_objects":42,"num_object_clones":0,"num_object_copies":126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":336,"num_write_kb":2688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":2359296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.2","version":"128'455","reported_seq":519,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-22T03:18:26.347752+0000","last_change":"2024-09-22T03:18:01.847644+0000","last_active":"2024-09-22T03:18:26.347752+0000","last_peered":"2024-09-22T03:18:26.347752+0000","last_clean":"2024-09-22T03:18:26.347752+0000","last_became_active":"2024-09-22T03:18:01.225010+0000","last_became_peered":"2024-09-22T03:18:01.225010+0000","last_unstale":"2024-09-22T03:18:26.347752+0000","last_undegraded":"2024-09-22T03:18:26.347752+0000","last_fullsized":"2024-09-22T03:18:26.347752+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:17:25.017919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:17:25.017919+0000","last_clean_scrub_stamp":"2024-09-22T03:17:25.017919+0000","objects_scrubbed":0,"log_size":455,"log_dups_size":0,"ondisk_log_size":455,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3735552,"num_objects":57,"num_object_clones":0,"num_object_copies":171,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":456,"num_write_kb":3648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":393216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1","version":"127'480","reported_seq":590,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-22T03:18:24.939913+0000","last_change":"2024-09-22T03:18:23.981632+0000","last_active":"2024-09-22T03:18:24.939913+0000","last_peered":"2024-09-22T03:18:24.939913+0000","last_clean":"2024-09-22T03:18:24.939913+0000","last_became_active":"2024-09-22T03:18:23.978972+0000","last_became_peered":"2024-09-22T03:18:23.978972+0000","last_unstale":"2024-09-22T03:18:24.939913+0000","last_undegraded":"2024-09-22T03:18:24.939913+0000","last_fullsized":"2024-09-22T03:18:24.939913+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:17:25.017919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:17:25.017919+0000","last_clean_scrub_stamp":"2024-09-22T03:17:25.017919+0000","objects_scrubbed":0,"log_size":480,"log_dups_size":0,"ondisk_log_size":480,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3932160,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":480,"num_write_kb":3840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":3342336,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"128'399","reported_seq":494,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-22T03:18:27.075396+0000","last_change":"2024-09-22T03:18:22.112809+0000","last_active":"2024-09-22T03:18:27.075396+0000","last_peered":"2024-09-22T03:18:27.075396+0000","last_clean":"2024-09-22T03:18:27.075396+0000","last_became_active":"2024-09-22T03:18:22.107439+0000","last_became_peered":"2024-09-22T03:18:22.107439+0000","last_unstale":"2024-09-22T03:18:27.075396+0000","last_undegraded":"2024-09-22T03:18:27.075396+0000","last_fullsized":"2024-09-22T03:18:27.075396+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:17:25.017919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:17:25.017919+0000","last_clean_scrub_stamp":"2024-09-22T03:17:25.017919+0000","objects_scrubbed":0,"log_size":399,"log_dups_size":0,"ondisk_log_size":399,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3276800,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":400,"num_write_kb":3200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":2490368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.9","version":"128'495","reported_seq":556,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-22T03:18:26.868930+0000","last_change":"2024-09-22T03:18:01.629983+0000","last_active":"2024-09-22T03:18:26.868930+0000","last_peered":"2024-09-22T03:18:26.868930+0000","last_clean":"2024-09-22T03:18:26.868930+0000","last_became_active":"2024-09-22T03:18:01.196590+0000","last_became_peered":"2024-09-22T03:18:01.196590+0000","last_unstale":"2024-09-22T03:18:26.868930+0000","last_undegraded":"2024-09-22T03:18:26.868930+0000","last_fullsized":"2024-09-22T03:18:26.868930+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:17:25.017919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:17:25.017919+0000","last_clean_scrub_stamp":"2024-09-22T03:17:25.017919+0000","objects_scrubbed":0,"log_size":495,"log_dups_size":0,"ondisk_log_size":495,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4063232,"num_objects":62,"num_object_clones":0,"num_object_copies":186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":496,"num_write_kb":3968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":393216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":176,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-22T03:18:24.940295+0000","last_change":"2024-09-22T03:18:17.050118+0000","last_active":"2024-09-22T03:18:24.940295+0000","last_peered":"2024-09-22T03:18:24.940295+0000","last_clean":"2024-09-22T03:18:24.940295+0000","last_became_active":"2024-09-22T03:18:01.415712+0000","last_became_peered":"2024-09-22T03:18:01.415712+0000","last_unstale":"2024-09-22T03:18:24.940295+0000","last_undegraded":"2024-09-22T03:18:24.940295+0000","last_fullsized":"2024-09-22T03:18:24.940295+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.8","version":"128'416","reported_seq":565,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-22T03:18:26.882824+0000","last_change":"2024-09-22T03:18:22.093165+0000","last_active":"2024-09-22T03:18:26.882824+0000","last_peered":"2024-09-22T03:18:26.882824+0000","last_clean":"2024-09-22T03:18:26.882824+0000","last_became_active":"2024-09-22T03:18:22.084001+0000","last_became_peered":"2024-09-22T03:18:22.084001+0000","last_unstale":"2024-09-22T03:18:26.882824+0000","last_undegraded":"2024-09-22T03:18:26.882824+0000","last_fullsized":"2024-09-22T03:18:26.882824+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:17:25.017919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:17:25.017919+0000","last_clean_scrub_stamp":"2024-09-22T03:17:25.017919+0000","objects_scrubbed":0,"log_size":416,"log_dups_size":0,"ondisk_log_size":416,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3416064,"num_objects":53,"num_object_clones":0,"num_object_copies":159,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":417,"num_write_kb":3336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":2957312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":308,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-22T03:18:24.944346+0000","last_change":"2024-09-22T03:18:19.078674+0000","last_active":"2024-09-22T03:18:24.944346+0000","last_peered":"2024-09-22T03:18:24.944346+0000","last_clean":"2024-09-22T03:18:24.944346+0000","last_became_active":"2024-09-22T03:18:18.121868+0000","last_became_peered":"2024-09-22T03:18:18.121868+0000","last_unstale":"2024-09-22T03:18:24.944346+0000","last_undegraded":"2024-09-22T03:18:24.944346+0000","last_fullsized":"2024-09-22T03:18:24.944346+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.b","version":"128'327","reported_seq":441,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-22T03:18:26.342256+0000","last_change":"2024-09-22T03:18:23.986969+0000","last_active":"2024-09-22T03:18:26.342256+0000","last_peered":"2024-09-22T03:18:26.342256+0000","last_clean":"2024-09-22T03:18:26.342256+0000","last_became_active":"2024-09-22T03:18:23.981163+0000","last_became_peered":"2024-09-22T03:18:23.981163+0000","last_unstale":"2024-09-22T03:18:26.342256+0000","last_undegraded":"2024-09-22T03:18:26.342256+0000","last_fullsized":"2024-09-22T03:18:26.342256+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:17:25.017919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:17:25.017919+0000","last_clean_scrub_stamp":"2024-09-22T03:17:25.017919+0000","objects_scrubbed":0,"log_size":327,"log_dups_size":0,"ondisk_log_size":327,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2686976,"num_objects":41,"num_object_clones":0,"num_object_copies":123,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":328,"num_write_kb":2624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":2031616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":305,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-22T03:18:24.940266+0000","last_change":"2024-09-22T03:18:19.067090+0000","last_active":"2024-09-22T03:18:24.940266+0000","last_peered":"2024-09-22T03:18:24.940266+0000","last_clean":"2024-09-22T03:18:24.940266+0000","last_became_active":"2024-09-22T03:18:18.104717+0000","last_became_peered":"2024-09-22T03:18:18.104717+0000","last_unstale":"2024-09-22T03:18:24.940266+0000","last_undegraded":"2024-09-22T03:18:24.940266+0000","last_fullsized":"2024-09-22T03:18:24.940266+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.a","version":"127'336","reported_seq":437,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-22T03:18:24.941405+0000","last_change":"2024-09-22T03:18:23.099898+0000","last_active":"2024-09-22T03:18:24.941405+0000","last_peered":"2024-09-22T03:18:24.941405+0000","last_clean":"2024-09-22T03:18:24.941405+0000","last_became_active":"2024-09-22T03:18:23.094343+0000","last_became_peered":"2024-09-22T03:18:23.094343+0000","last_unstale":"2024-09-22T03:18:24.941405+0000","last_undegraded":"2024-09-22T03:18:24.941405+0000","last_fullsized":"2024-09-22T03:18:24.941405+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:17:25.017919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:17:25.017919+0000","last_clean_scrub_stamp":"2024-09-22T03:17:25.017919+0000","objects_scrubbed":0,"log_size":336,"log_dups_size":0,"ondisk_log_size":336,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2752512,"num_objects":42,"num_object_clones":0,"num_object_copies":126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":336,"num_write_kb":2688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39,"num_bytes_recovered":2408448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":198,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-22T03:18:24.938427+0000","last_change":"2024-09-22T03:18:19.050753+0000","last_active":"2024-09-22T03:18:24.938427+0000","last_peered":"2024-09-22T03:18:24.938427+0000","last_clean":"2024-09-22T03:18:24.938427+0000","last_became_active":"2024-09-22T03:18:18.099051+0000","last_became_peered":"2024-09-22T03:18:18.099051+0000","last_unstale":"2024-09-22T03:18:24.938427+0000","last_undegraded":"2024-09-22T03:18:24.938427+0000","last_fullsized":"2024-09-22T03:18:24.938427+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"12'97","reported_seq":429,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-22T03:18:24.948002+0000","last_change":"2024-09-22T03:18:18.126160+0000","last_active":"2024-09-22T03:18:24.948002+0000","last_peered":"2024-09-22T03:18:24.948002+0000","last_clean":"2024-09-22T03:18:24.948002+0000","last_became_active":"2024-09-22T03:18:18.122292+0000","last_became_peered":"2024-09-22T03:18:18.122292+0000","last_unstale":"2024-09-22T03:18:24.948002+0000","last_undegraded":"2024-09-22T03:18:24.948002+0000","last_fullsized":"2024-09-22T03:18:24.948002+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.d","version":"126'336","reported_seq":458,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-22T03:18:24.993358+0000","last_change":"2024-09-22T03:18:24.994136+0000","last_active":"2024-09-22T03:18:24.993358+0000","last_peered":"2024-09-22T03:18:24.993358+0000","last_clean":"2024-09-22T03:17:28.160849+0000","last_became_active":"2024-09-22T03:18:24.975245+0000","last_became_peered":"2024-09-22T03:18:24.975245+0000","last_unstale":"2024-09-22T03:18:24.993358+0000","last_undegraded":"2024-09-22T03:18:24.993358+0000","last_fullsized":"2024-09-22T03:18:24.993358+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:17:25.017919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:17:25.017919+0000","last_clean_scrub_stamp":"2024-09-22T03:17:25.017919+0000","objects_scrubbed":0,"log_size":336,"log_dups_size":0,"ondisk_log_size":336,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2752512,"num_objects":42,"num_object_clones":0,"num_object_copies":126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":336,"num_write_kb":2688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":2359296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":258,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-22T03:18:24.941872+0000","last_change":"2024-09-22T03:18:19.044456+0000","last_active":"2024-09-22T03:18:24.941872+0000","last_peered":"2024-09-22T03:18:24.941872+0000","last_clean":"2024-09-22T03:18:24.941872+0000","last_became_active":"2024-09-22T03:18:18.129858+0000","last_became_peered":"2024-09-22T03:18:18.129858+0000","last_unstale":"2024-09-22T03:18:24.941872+0000","last_undegraded":"2024-09-22T03:18:24.941872+0000","last_fullsized":"2024-09-22T03:18:24.941872+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.c","version":"128'513","reported_seq":668,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-22T03:18:26.908094+0000","last_change":"2024-09-22T03:18:24.986791+0000","last_active":"2024-09-22T03:18:26.908094+0000","last_peered":"2024-09-22T03:18:26.908094+0000","last_clean":"2024-09-22T03:18:26.908094+0000","last_became_active":"2024-09-22T03:18:24.966848+0000","last_became_peered":"2024-09-22T03:18:24.966848+0000","last_unstale":"2024-09-22T03:18:26.908094+0000","last_undegraded":"2024-09-22T03:18:26.908094+0000","last_fullsized":"2024-09-22T03:18:26.908094+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:17:25.017919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:17:25.017919+0000","last_clean_scrub_stamp":"2024-09-22T03:17:25.017919+0000","objects_scrubbed":0,"log_size":513,"log_dups_size":0,"ondisk_log_size":513,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4210688,"num_objects":65,"num_object_clones":0,"num_object_copies":195,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":514,"num_write_kb":4112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":51,"num_bytes_recovered":3342336,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":280,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-22T03:18:24.940475+0000","last_change":"2024-09-22T03:18:17.081823+0000","last_active":"2024-09-22T03:18:24.940475+0000","last_peered":"2024-09-22T03:18:24.940475+0000","last_clean":"2024-09-22T03:18:24.940475+0000","last_became_active":"2024-09-22T03:18:01.232732+0000","last_became_peered":"2024-09-22T03:18:01.232732+0000","last_unstale":"2024-09-22T03:18:24.940475+0000","last_undegraded":"2024-09-22T03:18:24.940475+0000","last_fullsized":"2024-09-22T03:18:24.940475+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.f","version":"128'431","reported_seq":557,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-22T03:18:25.305301+0000","last_change":"2024-09-22T03:18:23.087718+0000","last_active":"2024-09-22T03:18:25.305301+0000","last_peered":"2024-09-22T03:18:25.305301+0000","last_clean":"2024-09-22T03:18:25.305301+0000","last_became_active":"2024-09-22T03:18:23.080292+0000","last_became_peered":"2024-09-22T03:18:23.080292+0000","last_unstale":"2024-09-22T03:18:25.305301+0000","last_undegraded":"2024-09-22T03:18:25.305301+0000","last_fullsized":"2024-09-22T03:18:25.305301+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:17:25.017919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:17:25.017919+0000","last_clean_scrub_stamp":"2024-09-22T03:17:25.017919+0000","objects_scrubbed":0,"log_size":431,"log_dups_size":0,"ondisk_log_size":431,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3538944,"num_objects":54,"num_object_clones":0,"num_object_copies":162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":432,"num_write_kb":3456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":3014656,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":306,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-22T03:18:24.937639+0000","last_change":"2024-09-22T03:18:19.043166+0000","last_active":"2024-09-22T03:18:24.937639+0000","last_peered":"2024-09-22T03:18:24.937639+0000","last_clean":"2024-09-22T03:18:24.937639+0000","last_became_active":"2024-09-22T03:18:18.098084+0000","last_became_peered":"2024-09-22T03:18:18.098084+0000","last_unstale":"2024-09-22T03:18:24.937639+0000","last_undegraded":"2024-09-22T03:18:24.937639+0000","last_fullsized":"2024-09-22T03:18:24.937639+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.e","version":"128'407","reported_seq":544,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-22T03:18:26.545751+0000","last_change":"2024-09-22T03:18:24.972978+0000","last_active":"2024-09-22T03:18:26.545751+0000","last_peered":"2024-09-22T03:18:26.545751+0000","last_clean":"2024-09-22T03:18:26.545751+0000","last_became_active":"2024-09-22T03:18:24.969080+0000","last_became_peered":"2024-09-22T03:18:24.969080+0000","last_unstale":"2024-09-22T03:18:26.545751+0000","last_undegraded":"2024-09-22T03:18:26.545751+0000","last_fullsized":"2024-09-22T03:18:26.545751+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:17:25.017919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:17:25.017919+0000","last_clean_scrub_stamp":"2024-09-22T03:17:25.017919+0000","objects_scrubbed":0,"log_size":407,"log_dups_size":0,"ondisk_log_size":407,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3342336,"num_objects":51,"num_object_clones":0,"num_object_copies":153,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":408,"num_write_kb":3264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":42,"num_bytes_recovered":2752512,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":299,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-22T03:18:24.938752+0000","last_change":"2024-09-22T03:18:19.042797+0000","last_active":"2024-09-22T03:18:24.938752+0000","last_peered":"2024-09-22T03:18:24.938752+0000","last_clean":"2024-09-22T03:18:24.938752+0000","last_became_active":"2024-09-22T03:18:18.118143+0000","last_became_peered":"2024-09-22T03:18:18.118143+0000","last_unstale":"2024-09-22T03:18:24.938752+0000","last_undegraded":"2024-09-22T03:18:24.938752+0000","last_fullsized":"2024-09-22T03:18:24.938752+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":56246272,"num_objects":861,"num_object_clones":0,"num_object_copies":2583,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6866,"num_write_kb":54928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":629,"num_bytes_recovered":40198144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":66355200,"data_stored":168706048,"data_compressed":8549059,"data_compressed_allocated":66355200,"data_compressed_original":168706048,"omap_allocated":0,"internal_metadata":0},"log_size":6853,"ondisk_log_size":6853,"up":48,"acting":48,"num_store_stats":4},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1180736,"data_compressed":59924,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":119,"seq":511101108249,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":50740,"kb_used_data":19812,"kb_used_omap":0,"kb_used_meta":30912,"kb_avail":94321100,"statfs":{"total":96636764160,"available":96584806400,"internally_reserved":0,"allocated":20287488,"data_stored":41261189,"data_compressed":2097087,"data_compressed_allocated":19210240,"data_compressed_original":40828928,"omap_allocated":0,"internal_metadata":31653888},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":100,"seq":429496729729,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":57032,"kb_used_data":16120,"kb_used_omap":0,"kb_used_meta":40896,"kb_avail":94314808,"statfs":{"total":96636764160,"available":96578363392,"internally_reserved":0,"allocated":16506880,"data_stored":31759013,"data_compressed":1616406,"data_compressed_allocated":15409152,"data_compressed_original":31309824,"omap_allocated":0,"internal_metadata":41877504},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":121,"seq":519691042826,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":50540,"kb_used_data":8860,"kb_used_omap":0,"kb_used_meta":41664,"kb_avail":94321300,"statfs":{"total":96636764160,"available":96585011200,"internally_reserved":0,"allocated":9072640,"data_stored":50453157,"data_compressed":2432991,"data_compressed_allocated":7974912,"data_compressed_original":50003968,"omap_allocated":0,"internal_metadata":42663936},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":73,"seq":313532612838,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":77336,"kb_used_data":22984,"kb_used_omap":0,"kb_used_meta":54336,"kb_avail":94294504,"statfs":{"total":96636764160,"available":96557572096,"internally_reserved":0,"allocated":23535616,"data_stored":45849234,"data_compressed":2343565,"data_compressed_allocated":22441984,"data_compressed_original":45400064,"omap_allocated":0,"internal_metadata":55640064},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29962,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29962,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":22904832,"data_stored":45834240,"data_compressed":2365632,"data_compressed_allocated":22904832,"data_compressed_original":45834240,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8105984,"data_stored":49774592,"data_compressed":2420795,"data_compressed_allocated":8105984,"data_compressed_original":49774592,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":15872000,"data_stored":31744000,"data_compressed":1638473,"data_compressed_allocated":15872000,"data_compressed_original":31744000,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":19472384,"data_stored":41353216,"data_compressed":2124159,"data_compressed_allocated":19472384,"data_compressed_original":41353216,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-22T03:18:30.069 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-09-22T03:18:30.069 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-22T03:18:30.069 INFO:tasks.thrashosds.thrasher:Killing osd 3, live_osds are [0, 2, 3, 1] 2024-09-22T03:18:30.070 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-09-22T03:18:30.070 INFO:teuthology.orchestra.run:waiting for 300 2024-09-22T03:18:30.112 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 63 16 7935 7919 0.981909 2.3125 0.0225819 0.127184 2024-09-22T03:18:31.112 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 64 16 7936 7920 0.966688 0.0078125 0.0145684 0.127169 2024-09-22T03:18:31.876 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-22T03:18:31.995 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:18:31.991+0000 7f46c6917640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:18:31.998 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:18:31.996+0000 7f46c6917640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:18:32.112 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 65 16 7936 7920 0.951816 0 - 0.127169 2024-09-22T03:18:32.686 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~0s 2024-09-22T03:18:33.112 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 66 16 7936 7920 0.937394 0 - 0.127169 2024-09-22T03:18:34.112 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 67 16 7936 7920 0.923403 0 - 0.127169 2024-09-22T03:18:35.113 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 68 16 7936 7920 0.909823 0 - 0.127169 2024-09-22T03:18:36.113 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 69 16 7936 7920 0.896637 0 - 0.127169 2024-09-22T03:18:36.172 INFO:tasks.ceph.osd.3:Stopped 2024-09-22T03:18:37.113 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 70 16 7936 7920 0.883827 0 - 0.127169 2024-09-22T03:18:38.113 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 71 16 7936 7920 0.871379 0 - 0.127169 2024-09-22T03:18:38.191 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 has been restored 2024-09-22T03:18:39.113 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 72 16 7936 7920 0.859276 0 - 0.127169 2024-09-22T03:18:40.113 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 73 16 7936 7920 0.847505 0 - 0.127169 2024-09-22T03:18:41.113 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 74 16 7936 7920 0.836052 0 - 0.127169 2024-09-22T03:18:41.173 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [3] live_osds: [0, 2, 1] 2024-09-22T03:18:41.173 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-22T03:18:41.173 INFO:tasks.thrashosds.thrasher:Reviving osd 3 2024-09-22T03:18:41.173 INFO:tasks.ceph.osd.3:Restarting daemon 2024-09-22T03:18:41.174 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2>> /var/log/ceph/ceph-osd.3.log 2024-09-22T03:18:41.176 INFO:tasks.ceph.osd.3:Started 2024-09-22T03:18:41.177 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-22T03:18:41.287 INFO:teuthology.orchestra.run.smithi196.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-09-22T03:18:41.290 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-22T03:18:41.290 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-09-22T03:18:42.113 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 75 16 7936 7920 0.824905 0 - 0.127169 2024-09-22T03:18:43.114 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 76 16 7936 7920 0.81405 0 - 0.127169 2024-09-22T03:18:44.114 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 77 16 7936 7920 0.803478 0 - 0.127169 2024-09-22T03:18:45.114 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 78 16 7936 7920 0.793177 0 - 0.127169 2024-09-22T03:18:46.114 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 79 16 7936 7920 0.783136 0 - 0.127169 2024-09-22T03:18:46.291 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-22T03:18:46.387 INFO:teuthology.orchestra.run.smithi196.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-09-22T03:18:46.389 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-22T03:18:46.390 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-09-22T03:18:47.114 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:2024-09-22T03:18:47.113841+0000 min lat: 0.00665283 max lat: 33.1634 avg lat: 0.127169 2024-09-22T03:18:47.114 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-22T03:18:47.114 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 80 16 7936 7920 0.773346 0 - 0.127169 2024-09-22T03:18:48.114 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 81 16 7936 7920 0.763799 0 - 0.127169 2024-09-22T03:18:49.115 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 82 16 7936 7920 0.754484 0 - 0.127169 2024-09-22T03:18:49.938 INFO:teuthology.orchestra.run.smithi153.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-09-22T03:18:49.943 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-22T03:18:49.944 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-22T03:18:50.115 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 83 16 7936 7920 0.745393 0 - 0.127169 2024-09-22T03:18:51.115 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 84 16 7936 7920 0.73652 0 - 0.127169 2024-09-22T03:18:51.391 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-22T03:18:51.502 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-09-22T03:18:51.502 INFO:teuthology.orchestra.run.smithi196.stdout: "ops_in_flight": [ 2024-09-22T03:18:51.502 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-22T03:18:51.502 INFO:teuthology.orchestra.run.smithi196.stdout: "type": "client_request", 2024-09-22T03:18:51.502 INFO:teuthology.orchestra.run.smithi196.stdout: "id": 61, 2024-09-22T03:18:51.502 INFO:teuthology.orchestra.run.smithi196.stdout: "detail": { 2024-09-22T03:18:51.502 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:18:51.503 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:18:51.503 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:18:50.997989+0000" 2024-09-22T03:18:51.503 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.503 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:18:51.503 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:18:51.503 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:18:50.998008+0000" 2024-09-22T03:18:51.503 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.503 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:18:51.503 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:18:51.503 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.503 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.503 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:18:51.504 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:18:51.504 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:18:50.998190+0000" 2024-09-22T03:18:51.504 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.504 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:18:51.504 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:18:51.504 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.504 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.504 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:18:51.504 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:18:51.504 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.504 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.504 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:18:51.504 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:18:51.504 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.505 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.505 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:18:51.505 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:18:51.505 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:18:50.998681+0000" 2024-09-22T03:18:51.505 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.505 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:18:51.505 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:18:51.505 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.505 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.505 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:18:51.505 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:18:51.506 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:18:50.999561+0000" 2024-09-22T03:18:51.506 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.506 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:18:51.506 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:18:51.506 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.506 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.506 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:18:51.506 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:18:51.506 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.506 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.506 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:18:51.506 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:18:51.506 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.506 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.507 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:18:51.507 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:18:51.507 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.507 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.507 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:18:51.507 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:18:51.507 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:18:50.999825+0000" 2024-09-22T03:18:51.507 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.507 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:18:51.507 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:18:51.507 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.507 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.507 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:18:51.507 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:18:51.508 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.508 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.508 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:18:51.508 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:18:51.508 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.508 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.508 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:18:51.508 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:18:51.508 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:18:50.999968+0000" 2024-09-22T03:18:51.508 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.509 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:18:51.509 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:18:51.509 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.509 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.509 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:18:51.509 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:18:51.509 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:18:51.000054+0000", 2024-09-22T03:18:51.510 INFO:teuthology.orchestra.run.smithi196.stdout: "blocker": { 2024-09-22T03:18:51.510 INFO:teuthology.orchestra.run.smithi196.stdout: "op_type": "PGActivationBlocker", 2024-09-22T03:18:51.510 INFO:teuthology.orchestra.run.smithi196.stdout: "detail": { 2024-09-22T03:18:51.510 INFO:teuthology.orchestra.run.smithi196.stdout: "pgid": "8.f" 2024-09-22T03:18:51.510 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:18:51.510 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:18:51.511 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.511 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:18:51.511 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:18:51.511 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.511 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.511 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:18:51.511 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:18:51.512 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.512 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.512 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:18:51.512 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:18:51.512 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.512 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.512 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:18:51.512 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:18:51.513 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.513 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.513 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:18:51.513 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:18:51.513 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.513 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.513 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:18:51.514 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:18:51.514 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.514 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.514 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:18:51.514 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:18:51.514 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.514 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.515 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:18:51.515 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:18:51.515 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.515 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.515 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:18:51.515 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:18:51.516 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.516 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.516 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:18:51.516 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:18:51.516 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.516 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.516 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:18:51.516 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:18:51.516 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.517 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.517 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:18:51.517 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:18:51.517 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.517 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.517 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:18:51.517 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:18:51.517 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.517 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.517 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:18:51.517 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:18:51.517 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.517 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.518 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:18:51.518 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:18:51.518 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.518 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.518 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:18:51.518 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:18:51.518 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.518 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.518 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:18:51.518 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:18:51.518 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.518 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.518 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:18:51.519 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:18:51.519 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.519 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.519 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:18:51.519 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:18:51.519 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.519 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.519 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:18:51.519 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:18:51.519 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.519 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:18:51.519 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:18:51.519 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.520 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-22T03:18:51.520 INFO:teuthology.orchestra.run.smithi196.stdout: "type": "client_request", 2024-09-22T03:18:51.520 INFO:teuthology.orchestra.run.smithi196.stdout: "id": 65, 2024-09-22T03:18:51.520 INFO:teuthology.orchestra.run.smithi196.stdout: "detail": { 2024-09-22T03:18:51.520 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:18:51.520 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:18:51.520 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:18:51.000556+0000" 2024-09-22T03:18:51.520 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.520 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:18:51.520 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:18:51.520 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:18:51.000560+0000" 2024-09-22T03:18:51.520 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.520 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:18:51.520 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:18:51.521 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.521 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.521 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:18:51.521 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:18:51.521 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:18:51.000807+0000" 2024-09-22T03:18:51.521 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.521 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:18:51.521 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:18:51.521 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.521 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.521 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:18:51.521 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:18:51.521 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.521 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.521 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:18:51.522 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:18:51.522 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.522 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.522 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:18:51.522 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:18:51.522 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:18:51.000885+0000" 2024-09-22T03:18:51.522 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.522 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:18:51.522 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:18:51.522 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.522 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.522 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:18:51.522 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:18:51.523 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:18:51.001527+0000" 2024-09-22T03:18:51.523 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.523 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:18:51.523 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:18:51.523 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.523 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.523 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:18:51.523 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:18:51.523 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.523 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.523 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:18:51.523 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:18:51.523 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.523 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.524 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:18:51.524 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:18:51.524 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.524 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.524 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:18:51.524 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:18:51.524 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:18:51.001753+0000" 2024-09-22T03:18:51.524 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.524 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:18:51.524 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:18:51.524 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.524 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.524 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:18:51.524 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:18:51.525 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.525 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.525 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:18:51.525 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:18:51.525 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.525 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.525 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:18:51.525 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:18:51.525 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:18:51.001862+0000", 2024-09-22T03:18:51.525 INFO:teuthology.orchestra.run.smithi196.stdout: "blocker": { 2024-09-22T03:18:51.525 INFO:teuthology.orchestra.run.smithi196.stdout: "op_type": "CommonPGPipeline:::wait_for_active", 2024-09-22T03:18:51.525 INFO:teuthology.orchestra.run.smithi196.stdout: "detail": { 2024-09-22T03:18:51.525 INFO:teuthology.orchestra.run.smithi196.stdout: "waiting": 1, 2024-09-22T03:18:51.525 INFO:teuthology.orchestra.run.smithi196.stdout: "held_by_operation_id": 61 2024-09-22T03:18:51.525 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:18:51.526 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:18:51.526 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.526 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:18:51.526 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:18:51.526 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.526 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.526 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:18:51.526 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:18:51.526 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.526 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.526 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:18:51.526 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:18:51.526 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.527 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.527 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:18:51.527 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:18:51.527 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.527 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.527 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:18:51.527 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:18:51.527 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.527 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.527 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:18:51.527 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:18:51.527 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.527 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.528 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:18:51.528 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:18:51.528 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.528 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.528 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:18:51.528 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:18:51.528 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.528 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.528 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:18:51.528 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:18:51.528 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.528 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.528 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:18:51.528 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:18:51.528 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.529 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.529 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:18:51.529 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:18:51.529 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.529 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.529 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:18:51.529 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:18:51.529 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.529 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.529 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:18:51.529 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:18:51.529 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.529 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.529 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:18:51.530 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:18:51.530 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.530 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.530 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:18:51.530 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:18:51.530 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.530 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.530 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:18:51.530 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:18:51.530 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.530 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.530 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:18:51.531 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:18:51.531 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.531 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.531 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:18:51.531 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:18:51.531 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.531 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.531 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:18:51.531 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:18:51.531 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.531 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.531 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:18:51.531 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:18:51.531 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.532 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.532 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:18:51.532 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:18:51.532 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.532 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.532 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:18:51.532 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:18:51.532 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.532 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:18:51.532 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:18:51.532 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.532 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-22T03:18:51.532 INFO:teuthology.orchestra.run.smithi196.stdout: "type": "client_request", 2024-09-22T03:18:51.533 INFO:teuthology.orchestra.run.smithi196.stdout: "id": 67, 2024-09-22T03:18:51.533 INFO:teuthology.orchestra.run.smithi196.stdout: "detail": { 2024-09-22T03:18:51.533 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:18:51.533 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:18:51.533 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:18:51.001871+0000" 2024-09-22T03:18:51.533 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.533 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:18:51.533 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:18:51.533 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:18:51.001875+0000" 2024-09-22T03:18:51.533 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.533 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:18:51.533 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:18:51.533 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.533 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.534 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:18:51.534 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:18:51.534 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:18:51.002310+0000" 2024-09-22T03:18:51.534 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.534 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:18:51.534 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:18:51.534 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.534 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.534 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:18:51.534 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:18:51.534 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.534 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.534 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:18:51.535 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:18:51.535 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.535 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.535 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:18:51.535 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:18:51.535 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:18:51.002411+0000" 2024-09-22T03:18:51.535 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.535 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:18:51.535 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:18:51.535 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.536 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.536 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:18:51.536 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:18:51.536 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:18:51.003008+0000" 2024-09-22T03:18:51.536 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.536 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:18:51.536 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:18:51.536 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.536 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.536 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:18:51.536 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:18:51.536 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.536 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.536 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:18:51.537 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:18:51.537 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.537 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.537 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:18:51.537 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:18:51.537 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.537 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.537 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:18:51.537 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:18:51.537 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:18:51.003265+0000", 2024-09-22T03:18:51.537 INFO:teuthology.orchestra.run.smithi196.stdout: "blocker": { 2024-09-22T03:18:51.537 INFO:teuthology.orchestra.run.smithi196.stdout: "op_type": "ClientRequest::PGPipeline::await_map", 2024-09-22T03:18:51.537 INFO:teuthology.orchestra.run.smithi196.stdout: "detail": { 2024-09-22T03:18:51.537 INFO:teuthology.orchestra.run.smithi196.stdout: "waiting": 1, 2024-09-22T03:18:51.538 INFO:teuthology.orchestra.run.smithi196.stdout: "held_by_operation_id": 65 2024-09-22T03:18:51.538 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:18:51.538 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:18:51.538 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.538 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:18:51.538 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:18:51.538 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.538 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.538 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:18:51.538 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:18:51.538 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.538 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.538 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:18:51.539 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:18:51.539 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.539 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.539 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:18:51.539 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:18:51.539 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.539 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.539 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:18:51.539 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:18:51.539 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.539 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.539 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:18:51.539 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:18:51.540 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.540 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.540 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:18:51.540 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:18:51.540 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.540 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.540 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:18:51.540 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:18:51.540 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.540 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.540 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:18:51.540 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:18:51.541 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.541 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.541 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:18:51.541 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:18:51.541 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.541 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.541 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:18:51.541 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:18:51.541 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.541 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.541 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:18:51.541 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:18:51.541 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.542 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.542 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:18:51.542 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:18:51.542 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.542 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.542 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:18:51.542 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:18:51.542 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.542 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.542 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:18:51.542 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:18:51.542 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.542 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.543 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:18:51.543 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:18:51.543 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.543 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.543 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:18:51.543 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:18:51.543 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.543 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.543 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:18:51.543 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:18:51.544 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.544 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.544 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:18:51.544 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:18:51.544 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.544 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.544 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:18:51.544 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:18:51.544 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.544 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.544 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:18:51.544 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:18:51.544 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.544 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.545 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:18:51.545 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:18:51.545 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.545 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.545 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:18:51.545 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:18:51.545 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.545 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.545 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:18:51.545 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:18:51.545 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.545 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.545 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:18:51.546 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:18:51.546 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.546 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.546 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:18:51.546 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:18:51.546 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.546 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:18:51.546 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:18:51.546 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.546 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-22T03:18:51.546 INFO:teuthology.orchestra.run.smithi196.stdout: "type": "client_request", 2024-09-22T03:18:51.546 INFO:teuthology.orchestra.run.smithi196.stdout: "id": 69, 2024-09-22T03:18:51.546 INFO:teuthology.orchestra.run.smithi196.stdout: "detail": { 2024-09-22T03:18:51.547 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:18:51.547 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:18:51.547 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:18:51.003211+0000" 2024-09-22T03:18:51.547 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.547 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:18:51.547 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:18:51.547 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:18:51.003216+0000" 2024-09-22T03:18:51.547 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.547 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:18:51.547 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:18:51.547 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.547 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.547 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:18:51.547 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:18:51.548 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:18:51.003434+0000" 2024-09-22T03:18:51.548 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.548 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:18:51.548 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:18:51.548 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.548 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.548 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:18:51.548 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:18:51.548 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.548 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.548 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:18:51.549 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:18:51.549 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.549 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.549 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:18:51.549 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:18:51.549 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:18:51.003627+0000" 2024-09-22T03:18:51.549 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.549 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:18:51.549 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:18:51.549 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.549 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.549 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:18:51.549 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:18:51.549 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:18:51.004642+0000", 2024-09-22T03:18:51.550 INFO:teuthology.orchestra.run.smithi196.stdout: "blocker": { 2024-09-22T03:18:51.550 INFO:teuthology.orchestra.run.smithi196.stdout: "op_type": "PerShardPipeline::create_or_wait_pg", 2024-09-22T03:18:51.550 INFO:teuthology.orchestra.run.smithi196.stdout: "detail": { 2024-09-22T03:18:51.550 INFO:teuthology.orchestra.run.smithi196.stdout: "waiting": 5, 2024-09-22T03:18:51.550 INFO:teuthology.orchestra.run.smithi196.stdout: "held_by_operation_id": 67 2024-09-22T03:18:51.550 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:18:51.550 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:18:51.550 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.550 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:18:51.550 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:18:51.550 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.550 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.550 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:18:51.551 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:18:51.551 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.551 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.551 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:18:51.551 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:18:51.551 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.551 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.551 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:18:51.551 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:18:51.551 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.551 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.551 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:18:51.552 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:18:51.552 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.552 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.552 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:18:51.552 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:18:51.552 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.552 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.552 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:18:51.552 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:18:51.552 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.552 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.552 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:18:51.552 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:18:51.552 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.552 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.553 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:18:51.553 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:18:51.553 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.553 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.553 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:18:51.553 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:18:51.553 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.553 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.553 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:18:51.553 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:18:51.553 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.553 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.553 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:18:51.554 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:18:51.554 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.554 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.554 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:18:51.554 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:18:51.554 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.554 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.554 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:18:51.554 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:18:51.554 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.554 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.554 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:18:51.554 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:18:51.554 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.555 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.555 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:18:51.555 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:18:51.555 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.555 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.555 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:18:51.555 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:18:51.555 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.555 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.555 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:18:51.555 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:18:51.555 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.555 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.555 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:18:51.556 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:18:51.556 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.556 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.556 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:18:51.556 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:18:51.556 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.556 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.556 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:18:51.556 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:18:51.556 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.556 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.557 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:18:51.557 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:18:51.557 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.557 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.557 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:18:51.557 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:18:51.557 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.557 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.557 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:18:51.557 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:18:51.557 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.558 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.558 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:18:51.558 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:18:51.558 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.558 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.558 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:18:51.558 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:18:51.558 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.558 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.558 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:18:51.558 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:18:51.558 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.559 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.559 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:18:51.559 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:18:51.559 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.559 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.559 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:18:51.559 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:18:51.559 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.559 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.559 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:18:51.559 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:18:51.559 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.559 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.559 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:18:51.560 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:18:51.560 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.560 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:18:51.560 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:18:51.560 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.560 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-22T03:18:51.560 INFO:teuthology.orchestra.run.smithi196.stdout: "type": "client_request", 2024-09-22T03:18:51.560 INFO:teuthology.orchestra.run.smithi196.stdout: "id": 72, 2024-09-22T03:18:51.560 INFO:teuthology.orchestra.run.smithi196.stdout: "detail": { 2024-09-22T03:18:51.560 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:18:51.560 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:18:51.561 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:18:51.005089+0000" 2024-09-22T03:18:51.561 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.561 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:18:51.561 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:18:51.561 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:18:51.005093+0000" 2024-09-22T03:18:51.561 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.561 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:18:51.561 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:18:51.561 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.561 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.561 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:18:51.561 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:18:51.562 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:18:51.005240+0000" 2024-09-22T03:18:51.562 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.562 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:18:51.562 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:18:51.562 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.562 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.562 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:18:51.562 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:18:51.562 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.562 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.562 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:18:51.562 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:18:51.562 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.562 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.563 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:18:51.563 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:18:51.563 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:18:51.005503+0000" 2024-09-22T03:18:51.563 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.563 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:18:51.563 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:18:51.563 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.563 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.563 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:18:51.563 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:18:51.563 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:18:51.006070+0000", 2024-09-22T03:18:51.563 INFO:teuthology.orchestra.run.smithi196.stdout: "blocker": { 2024-09-22T03:18:51.563 INFO:teuthology.orchestra.run.smithi196.stdout: "op_type": "PerShardPipeline::create_or_wait_pg", 2024-09-22T03:18:51.564 INFO:teuthology.orchestra.run.smithi196.stdout: "detail": { 2024-09-22T03:18:51.564 INFO:teuthology.orchestra.run.smithi196.stdout: "waiting": 5, 2024-09-22T03:18:51.564 INFO:teuthology.orchestra.run.smithi196.stdout: "held_by_operation_id": 67 2024-09-22T03:18:51.564 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:18:51.564 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:18:51.564 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.564 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:18:51.564 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:18:51.564 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.564 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.564 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:18:51.564 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:18:51.564 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.564 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.565 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:18:51.565 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:18:51.565 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.565 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.565 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:18:51.565 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:18:51.565 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.565 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.565 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:18:51.565 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:18:51.565 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.565 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.565 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:18:51.565 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:18:51.566 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.566 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.566 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:18:51.566 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:18:51.566 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.566 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.566 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:18:51.566 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:18:51.566 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.566 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.566 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:18:51.566 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:18:51.566 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.566 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.567 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:18:51.567 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:18:51.567 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.567 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.567 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:18:51.567 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:18:51.567 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.567 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.567 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:18:51.567 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:18:51.567 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.567 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.568 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:18:51.568 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:18:51.568 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.568 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.568 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:18:51.568 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:18:51.568 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.568 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.568 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:18:51.568 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:18:51.568 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.568 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.568 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:18:51.569 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:18:51.569 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.569 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.569 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:18:51.569 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:18:51.569 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.569 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.569 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:18:51.569 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:18:51.569 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.569 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.569 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:18:51.569 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:18:51.569 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.570 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.570 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:18:51.570 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:18:51.570 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.570 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.570 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:18:51.570 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:18:51.570 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.570 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.570 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:18:51.570 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:18:51.570 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.570 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.570 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:18:51.571 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:18:51.571 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.571 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.571 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:18:51.571 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:18:51.571 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.571 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.571 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:18:51.571 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:18:51.571 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.571 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.571 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:18:51.571 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:18:51.571 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.571 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.572 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:18:51.572 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:18:51.572 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.572 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.572 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:18:51.572 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:18:51.572 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.572 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.572 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:18:51.572 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:18:51.572 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.572 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.572 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:18:51.572 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:18:51.572 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.573 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.573 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:18:51.573 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:18:51.573 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.573 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:18:51.573 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:18:51.573 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.573 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-22T03:18:51.573 INFO:teuthology.orchestra.run.smithi196.stdout: "type": "client_request", 2024-09-22T03:18:51.573 INFO:teuthology.orchestra.run.smithi196.stdout: "id": 75, 2024-09-22T03:18:51.573 INFO:teuthology.orchestra.run.smithi196.stdout: "detail": { 2024-09-22T03:18:51.573 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:18:51.573 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:18:51.573 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:18:51.006527+0000" 2024-09-22T03:18:51.574 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.574 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:18:51.574 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:18:51.574 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:18:51.006531+0000" 2024-09-22T03:18:51.574 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.574 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:18:51.574 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:18:51.574 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.574 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.574 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:18:51.574 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:18:51.574 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:18:51.006641+0000" 2024-09-22T03:18:51.574 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.574 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:18:51.575 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:18:51.575 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.575 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.575 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:18:51.575 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:18:51.575 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.575 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.575 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:18:51.575 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:18:51.575 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.575 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.575 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:18:51.576 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:18:51.576 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:18:51.006823+0000" 2024-09-22T03:18:51.576 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.576 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:18:51.576 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:18:51.576 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.576 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.576 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:18:51.576 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:18:51.576 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:18:51.007843+0000", 2024-09-22T03:18:51.576 INFO:teuthology.orchestra.run.smithi196.stdout: "blocker": { 2024-09-22T03:18:51.576 INFO:teuthology.orchestra.run.smithi196.stdout: "op_type": "PerShardPipeline::create_or_wait_pg", 2024-09-22T03:18:51.576 INFO:teuthology.orchestra.run.smithi196.stdout: "detail": { 2024-09-22T03:18:51.577 INFO:teuthology.orchestra.run.smithi196.stdout: "waiting": 5, 2024-09-22T03:18:51.577 INFO:teuthology.orchestra.run.smithi196.stdout: "held_by_operation_id": 67 2024-09-22T03:18:51.577 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:18:51.577 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:18:51.577 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.577 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:18:51.577 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:18:51.577 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.577 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.577 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:18:51.577 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:18:51.577 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.577 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.577 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:18:51.578 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:18:51.578 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.578 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.578 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:18:51.578 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:18:51.578 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.578 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.578 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:18:51.578 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:18:51.578 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.578 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.578 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:18:51.579 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:18:51.579 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.579 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.579 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:18:51.579 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:18:51.579 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.579 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.579 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:18:51.579 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:18:51.579 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.579 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.579 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:18:51.579 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:18:51.580 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.580 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.580 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:18:51.580 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:18:51.580 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.580 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.580 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:18:51.580 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:18:51.580 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.580 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.580 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:18:51.580 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:18:51.580 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.580 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.581 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:18:51.581 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:18:51.581 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.581 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.581 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:18:51.581 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:18:51.581 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.581 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.581 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:18:51.581 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:18:51.581 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.581 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.581 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:18:51.582 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:18:51.582 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.582 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.582 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:18:51.582 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:18:51.582 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.582 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.582 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:18:51.582 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:18:51.582 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.582 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.582 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:18:51.582 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:18:51.582 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.583 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.583 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:18:51.583 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:18:51.583 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.583 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.583 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:18:51.583 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:18:51.583 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.583 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.583 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:18:51.583 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:18:51.583 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.583 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.584 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:18:51.584 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:18:51.584 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.584 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.584 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:18:51.584 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:18:51.584 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.584 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.584 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:18:51.584 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:18:51.584 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.584 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.584 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:18:51.585 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:18:51.585 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.585 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.585 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:18:51.585 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:18:51.585 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.585 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.585 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:18:51.585 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:18:51.585 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.585 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.585 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:18:51.585 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:18:51.585 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.586 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.586 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:18:51.586 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:18:51.586 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.586 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.586 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:18:51.586 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:18:51.586 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.586 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:18:51.586 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:18:51.586 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.586 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-22T03:18:51.586 INFO:teuthology.orchestra.run.smithi196.stdout: "type": "client_request", 2024-09-22T03:18:51.586 INFO:teuthology.orchestra.run.smithi196.stdout: "id": 78, 2024-09-22T03:18:51.587 INFO:teuthology.orchestra.run.smithi196.stdout: "detail": { 2024-09-22T03:18:51.587 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:18:51.587 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:18:51.587 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:18:51.008317+0000" 2024-09-22T03:18:51.587 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.587 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:18:51.587 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:18:51.587 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:18:51.008321+0000" 2024-09-22T03:18:51.587 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.587 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:18:51.587 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:18:51.587 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.587 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.587 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:18:51.588 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:18:51.588 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:18:51.008454+0000" 2024-09-22T03:18:51.588 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.588 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:18:51.588 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:18:51.588 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.588 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.588 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:18:51.588 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:18:51.588 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.588 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.588 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:18:51.588 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:18:51.588 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.588 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.589 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:18:51.589 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:18:51.589 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:18:51.008666+0000" 2024-09-22T03:18:51.589 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.589 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:18:51.589 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:18:51.589 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.589 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.589 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:18:51.589 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:18:51.589 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:18:51.009199+0000", 2024-09-22T03:18:51.589 INFO:teuthology.orchestra.run.smithi196.stdout: "blocker": { 2024-09-22T03:18:51.589 INFO:teuthology.orchestra.run.smithi196.stdout: "op_type": "PerShardPipeline::create_or_wait_pg", 2024-09-22T03:18:51.589 INFO:teuthology.orchestra.run.smithi196.stdout: "detail": { 2024-09-22T03:18:51.589 INFO:teuthology.orchestra.run.smithi196.stdout: "waiting": 5, 2024-09-22T03:18:51.590 INFO:teuthology.orchestra.run.smithi196.stdout: "held_by_operation_id": 67 2024-09-22T03:18:51.590 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:18:51.590 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:18:51.590 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.590 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:18:51.590 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:18:51.590 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.590 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.590 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:18:51.590 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:18:51.590 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.590 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.590 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:18:51.590 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:18:51.591 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.591 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.591 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:18:51.591 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:18:51.591 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.591 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.591 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:18:51.591 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:18:51.591 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.591 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.591 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:18:51.591 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:18:51.592 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.592 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.592 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:18:51.592 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:18:51.592 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.592 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.592 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:18:51.592 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:18:51.592 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.592 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.592 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:18:51.592 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:18:51.592 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.592 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.592 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:18:51.592 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:18:51.593 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.593 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.593 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:18:51.593 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:18:51.593 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.593 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.593 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:18:51.593 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:18:51.593 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.593 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.593 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:18:51.593 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:18:51.593 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.593 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.593 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:18:51.593 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:18:51.594 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.594 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.594 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:18:51.594 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:18:51.594 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.594 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.594 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:18:51.594 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:18:51.594 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.594 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.595 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:18:51.595 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:18:51.595 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.595 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.595 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:18:51.595 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:18:51.595 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.595 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.595 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:18:51.595 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:18:51.595 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.595 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.595 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:18:51.595 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:18:51.596 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.596 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.596 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:18:51.596 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:18:51.596 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.596 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.596 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:18:51.596 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:18:51.596 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.596 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.596 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:18:51.596 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:18:51.596 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.596 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.597 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:18:51.597 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:18:51.597 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.597 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.597 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:18:51.597 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:18:51.597 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.597 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.597 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:18:51.597 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:18:51.597 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.597 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.597 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:18:51.597 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:18:51.598 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.598 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.598 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:18:51.598 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:18:51.598 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.598 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.598 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:18:51.598 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:18:51.598 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.598 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.599 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:18:51.599 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:18:51.599 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.599 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.599 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:18:51.599 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:18:51.599 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.599 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:18:51.599 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:18:51.599 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.599 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-22T03:18:51.600 INFO:teuthology.orchestra.run.smithi196.stdout: "type": "client_request", 2024-09-22T03:18:51.600 INFO:teuthology.orchestra.run.smithi196.stdout: "id": 81, 2024-09-22T03:18:51.600 INFO:teuthology.orchestra.run.smithi196.stdout: "detail": { 2024-09-22T03:18:51.600 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:18:51.600 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:18:51.600 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:18:51.009765+0000" 2024-09-22T03:18:51.600 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.600 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:18:51.600 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:18:51.600 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:18:51.009768+0000" 2024-09-22T03:18:51.600 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.600 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:18:51.600 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:18:51.600 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.601 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.601 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:18:51.601 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:18:51.601 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:18:51.009901+0000" 2024-09-22T03:18:51.601 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.601 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:18:51.601 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:18:51.601 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.601 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.601 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:18:51.601 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:18:51.601 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.601 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.601 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:18:51.601 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:18:51.602 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.602 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.602 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:18:51.602 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:18:51.602 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:18:51.010067+0000" 2024-09-22T03:18:51.602 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.602 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:18:51.602 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:18:51.602 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.602 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.602 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:18:51.602 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:18:51.602 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:18:51.010292+0000", 2024-09-22T03:18:51.602 INFO:teuthology.orchestra.run.smithi196.stdout: "blocker": { 2024-09-22T03:18:51.602 INFO:teuthology.orchestra.run.smithi196.stdout: "op_type": "PerShardPipeline::create_or_wait_pg", 2024-09-22T03:18:51.603 INFO:teuthology.orchestra.run.smithi196.stdout: "detail": { 2024-09-22T03:18:51.603 INFO:teuthology.orchestra.run.smithi196.stdout: "waiting": 5, 2024-09-22T03:18:51.603 INFO:teuthology.orchestra.run.smithi196.stdout: "held_by_operation_id": 67 2024-09-22T03:18:51.603 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:18:51.603 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:18:51.603 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.603 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:18:51.603 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:18:51.603 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.603 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.603 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:18:51.604 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:18:51.604 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.604 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.604 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:18:51.604 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:18:51.604 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.605 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.605 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:18:51.605 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:18:51.605 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.605 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.605 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:18:51.605 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:18:51.606 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.606 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.606 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:18:51.606 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:18:51.606 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.606 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.606 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:18:51.606 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:18:51.607 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.607 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.607 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:18:51.607 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:18:51.607 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.607 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.607 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:18:51.607 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:18:51.608 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.608 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.608 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:18:51.608 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:18:51.608 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.608 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.608 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:18:51.609 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:18:51.609 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.609 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.609 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:18:51.609 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:18:51.609 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.609 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.610 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:18:51.610 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:18:51.610 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.610 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.610 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:18:51.610 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:18:51.610 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.611 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.611 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:18:51.611 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:18:51.611 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.611 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.611 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:18:51.611 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:18:51.611 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.611 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.611 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:18:51.611 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:18:51.612 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.612 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.612 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:18:51.612 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:18:51.612 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.612 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.612 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:18:51.612 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:18:51.612 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.612 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.612 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:18:51.612 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:18:51.612 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.612 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.613 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:18:51.613 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:18:51.613 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.613 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.613 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:18:51.613 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:18:51.613 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.613 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.613 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:18:51.613 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:18:51.613 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.613 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.613 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:18:51.613 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:18:51.614 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.614 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.614 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:18:51.614 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:18:51.614 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.614 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.614 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:18:51.614 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:18:51.614 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.614 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.614 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:18:51.614 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:18:51.614 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.614 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.615 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:18:51.615 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:18:51.615 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.615 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.615 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:18:51.615 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:18:51.615 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.615 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.615 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:18:51.615 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:18:51.615 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.615 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:18:51.615 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:18:51.616 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:18:51.616 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:18:51.616 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:18:51.616 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:18:51.616 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:18:51.616 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-22T03:18:51.616 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-09-22T03:18:51.944 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-22T03:18:52.087 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:18:52.084+0000 7f47f7197640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:18:52.092 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:18:52.090+0000 7f47f7197640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:18:52.115 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 85 16 7945 7929 0.728682 0.00334821 22.0776 0.15203 2024-09-22T03:18:53.115 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 86 16 8248 8232 0.747732 2.36719 0.0434993 0.167129 2024-09-22T03:18:54.115 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 87 16 8547 8531 0.765984 2.33594 0.0379682 0.163152 2024-09-22T03:18:54.325 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-22T03:18:54.468 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:18:54.465+0000 7fa6ab4d7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:18:54.473 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:18:54.471+0000 7fa6ab4d7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:18:55.115 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 88 16 8837 8821 0.783023 2.26562 0.0753855 0.159548 2024-09-22T03:18:56.115 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 89 16 9141 9125 0.800907 2.375 0.0886334 0.156007 2024-09-22T03:18:56.617 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 2, 1, 3] 2024-09-22T03:18:56.617 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-22T03:18:56.618 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-09-22T03:18:56.618 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-09-22T03:18:56.662 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-22T03:18:56.751 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:18:56.744+0000 7fdf46797640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:18:56.756 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:18:56.754+0000 7fdf46797640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:18:56.809 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:18:56.806+0000 7fa04ef57640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:18:56.813 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:18:56.811+0000 7fa04ef57640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:18:56.865 INFO:teuthology.orchestra.run.smithi153.stderr:no valid command found; 10 closest matches: 2024-09-22T03:18:56.865 INFO:teuthology.orchestra.run.smithi153.stderr:config get 2024-09-22T03:18:56.865 INFO:teuthology.orchestra.run.smithi153.stderr:config help 2024-09-22T03:18:56.865 INFO:teuthology.orchestra.run.smithi153.stderr:config set ... 2024-09-22T03:18:56.865 INFO:teuthology.orchestra.run.smithi153.stderr:config show 2024-09-22T03:18:56.865 INFO:teuthology.orchestra.run.smithi153.stderr:deep_scrub 2024-09-22T03:18:56.865 INFO:teuthology.orchestra.run.smithi153.stderr:dump_historic_ops 2024-09-22T03:18:56.865 INFO:teuthology.orchestra.run.smithi153.stderr:dump_historic_slow_ops 2024-09-22T03:18:56.865 INFO:teuthology.orchestra.run.smithi153.stderr:dump_metrics [] 2024-09-22T03:18:56.865 INFO:teuthology.orchestra.run.smithi153.stderr:dump_ops_in_flight 2024-09-22T03:18:56.866 INFO:teuthology.orchestra.run.smithi153.stderr:dump_pgstate_history 2024-09-22T03:18:56.866 INFO:teuthology.orchestra.run.smithi153.stderr:Error EINVAL: invalid command 2024-09-22T03:18:56.868 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-22T03:18:56.868 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-22T03:18:56.869 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-09-22T03:18:57.012 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:18:57.004+0000 7ff067197640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:18:57.016 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:18:57.014+0000 7ff067197640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:18:57.094 INFO:teuthology.orchestra.run.smithi153.stderr:no valid command found; 10 closest matches: 2024-09-22T03:18:57.094 INFO:teuthology.orchestra.run.smithi153.stderr:config get 2024-09-22T03:18:57.094 INFO:teuthology.orchestra.run.smithi153.stderr:config help 2024-09-22T03:18:57.094 INFO:teuthology.orchestra.run.smithi153.stderr:config set ... 2024-09-22T03:18:57.094 INFO:teuthology.orchestra.run.smithi153.stderr:config show 2024-09-22T03:18:57.094 INFO:teuthology.orchestra.run.smithi153.stderr:deep_scrub 2024-09-22T03:18:57.094 INFO:teuthology.orchestra.run.smithi153.stderr:dump_historic_ops 2024-09-22T03:18:57.095 INFO:teuthology.orchestra.run.smithi153.stderr:dump_historic_slow_ops 2024-09-22T03:18:57.095 INFO:teuthology.orchestra.run.smithi153.stderr:dump_metrics [] 2024-09-22T03:18:57.095 INFO:teuthology.orchestra.run.smithi153.stderr:dump_ops_in_flight 2024-09-22T03:18:57.095 INFO:teuthology.orchestra.run.smithi153.stderr:dump_pgstate_history 2024-09-22T03:18:57.095 INFO:teuthology.orchestra.run.smithi153.stderr:Error EINVAL: invalid command 2024-09-22T03:18:57.098 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-22T03:18:57.098 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-22T03:18:57.098 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-09-22T03:18:57.115 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 90 13 9419 9406 0.816398 2.19531 0.0527243 0.153063 2024-09-22T03:18:57.223 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:18:57.217+0000 7ffb24717640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:18:57.226 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:18:57.224+0000 7ffb24717640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:18:57.238 INFO:teuthology.orchestra.run.smithi153.stderr:no valid command found; 10 closest matches: 2024-09-22T03:18:57.238 INFO:teuthology.orchestra.run.smithi153.stderr:config get 2024-09-22T03:18:57.238 INFO:teuthology.orchestra.run.smithi153.stderr:config help 2024-09-22T03:18:57.238 INFO:teuthology.orchestra.run.smithi153.stderr:config set ... 2024-09-22T03:18:57.238 INFO:teuthology.orchestra.run.smithi153.stderr:config show 2024-09-22T03:18:57.238 INFO:teuthology.orchestra.run.smithi153.stderr:deep_scrub 2024-09-22T03:18:57.238 INFO:teuthology.orchestra.run.smithi153.stderr:dump_historic_ops 2024-09-22T03:18:57.238 INFO:teuthology.orchestra.run.smithi153.stderr:dump_historic_slow_ops 2024-09-22T03:18:57.239 INFO:teuthology.orchestra.run.smithi153.stderr:dump_metrics [] 2024-09-22T03:18:57.239 INFO:teuthology.orchestra.run.smithi153.stderr:dump_ops_in_flight 2024-09-22T03:18:57.239 INFO:teuthology.orchestra.run.smithi153.stderr:dump_pgstate_history 2024-09-22T03:18:57.239 INFO:teuthology.orchestra.run.smithi153.stderr:Error EINVAL: invalid command 2024-09-22T03:18:57.240 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-22T03:18:57.241 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-22T03:18:57.241 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-09-22T03:18:57.347 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:18:57.341+0000 7fbf5ff97640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:18:57.350 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:18:57.348+0000 7fbf5ff97640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:18:57.361 INFO:teuthology.orchestra.run.smithi153.stderr:no valid command found; 10 closest matches: 2024-09-22T03:18:57.361 INFO:teuthology.orchestra.run.smithi153.stderr:config get 2024-09-22T03:18:57.362 INFO:teuthology.orchestra.run.smithi153.stderr:config help 2024-09-22T03:18:57.362 INFO:teuthology.orchestra.run.smithi153.stderr:config set ... 2024-09-22T03:18:57.362 INFO:teuthology.orchestra.run.smithi153.stderr:config show 2024-09-22T03:18:57.362 INFO:teuthology.orchestra.run.smithi153.stderr:deep_scrub 2024-09-22T03:18:57.362 INFO:teuthology.orchestra.run.smithi153.stderr:dump_historic_ops 2024-09-22T03:18:57.362 INFO:teuthology.orchestra.run.smithi153.stderr:dump_historic_slow_ops 2024-09-22T03:18:57.362 INFO:teuthology.orchestra.run.smithi153.stderr:dump_metrics [] 2024-09-22T03:18:57.362 INFO:teuthology.orchestra.run.smithi153.stderr:dump_ops_in_flight 2024-09-22T03:18:57.362 INFO:teuthology.orchestra.run.smithi153.stderr:dump_pgstate_history 2024-09-22T03:18:57.362 INFO:teuthology.orchestra.run.smithi153.stderr:Error EINVAL: invalid command 2024-09-22T03:18:57.364 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-22T03:18:57.364 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-22T03:18:58.115 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:Total time run: 90.0484 2024-09-22T03:18:58.116 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:Total writes made: 9419 2024-09-22T03:18:58.116 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:Write size: 8192 2024-09-22T03:18:58.116 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:Object size: 65536 2024-09-22T03:18:58.116 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:Bandwidth (MB/sec): 0.817182 2024-09-22T03:18:58.116 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:Stddev Bandwidth: 1.16588 2024-09-22T03:18:58.116 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:Max bandwidth (MB/sec): 3.375 2024-09-22T03:18:58.116 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:Min bandwidth (MB/sec): 0 2024-09-22T03:18:58.116 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:Average IOPS: 104 2024-09-22T03:18:58.116 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:Stddev IOPS: 149.24 2024-09-22T03:18:58.116 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:Max IOPS: 432 2024-09-22T03:18:58.116 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:Min IOPS: 0 2024-09-22T03:18:58.116 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:Average Latency(s): 0.152924 2024-09-22T03:18:58.116 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:Stddev Latency(s): 1.60077 2024-09-22T03:18:58.117 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:Max latency(s): 33.1634 2024-09-22T03:18:58.117 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:Min latency(s): 0.00665283 2024-09-22T03:18:58.133 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:Cleaning up (deleting benchmark objects) 2024-09-22T03:18:58.982 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-22T03:18:59.129 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:18:59.126+0000 7f5fbc46c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:18:59.134 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:18:59.132+0000 7f5fbc46c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:19:01.380 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-22T03:19:01.450 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:Removed 1178 objects 2024-09-22T03:19:01.450 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:Clean up completed and total clean up time :3.31702 2024-09-22T03:19:01.503 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:19:01.500+0000 7f3593797640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:19:01.506 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:19:01.505+0000 7f3593797640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:19:02.366 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 2, 1, 3] 2024-09-22T03:19:02.366 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-22T03:19:02.366 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-22T03:19:02.472 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:19:02.467+0000 7f9dd3597640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:19:02.475 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:19:02.473+0000 7f9dd3597640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:19:02.650 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-22T03:19:02.650 INFO:teuthology.orchestra.run.smithi153.stderr:dumped all 2024-09-22T03:19:02.662 INFO:teuthology.orchestra.run.smithi153.stdout:{"pg_ready":true,"pg_map":{"version":455,"stamp":"2024-09-22T03:19:02.247671+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3236403,"num_objects":45,"num_object_clones":0,"num_object_copies":135,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":104,"num_read_kb":92,"num_write":10697,"num_write_kb":77249,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":630,"num_bytes_recovered":40206336,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":10642,"ondisk_log_size":10642,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":182144,"kb_used_data":20096,"kb_used_omap":0,"kb_used_meta":161984,"kb_avail":377305216,"statfs":{"total":386547056640,"available":386360541184,"internally_reserved":0,"allocated":20578304,"data_stored":35456297,"data_compressed":1737048,"data_compressed_allocated":16072704,"data_compressed_original":33619968,"omap_allocated":0,"internal_metadata":165871616},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-59899904,"num_objects":-915,"num_object_clones":0,"num_object_copies":-2745,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2923,"num_write_kb":14273,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":8192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001990"},"pg_stats":[{"pgid":"8.7","version":"132'773","reported_seq":959,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-22T03:19:01.382446+0000","last_change":"2024-09-22T03:18:51.995815+0000","last_active":"2024-09-22T03:19:01.382446+0000","last_peered":"2024-09-22T03:19:01.382446+0000","last_clean":"2024-09-22T03:19:01.382446+0000","last_became_active":"2024-09-22T03:18:51.992769+0000","last_became_peered":"2024-09-22T03:18:51.992769+0000","last_unstale":"2024-09-22T03:19:01.382446+0000","last_undegraded":"2024-09-22T03:19:01.382446+0000","last_fullsized":"2024-09-22T03:19:01.382446+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:17:25.017919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:17:25.017919+0000","last_clean_scrub_stamp":"2024-09-22T03:17:25.017919+0000","objects_scrubbed":0,"log_size":773,"log_dups_size":0,"ondisk_log_size":773,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":774,"num_write_kb":5504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":3866624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.6","version":"132'575","reported_seq":711,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-22T03:19:01.317445+0000","last_change":"2024-09-22T03:18:23.108631+0000","last_active":"2024-09-22T03:19:01.317445+0000","last_peered":"2024-09-22T03:19:01.317445+0000","last_clean":"2024-09-22T03:19:01.317445+0000","last_became_active":"2024-09-22T03:18:23.103912+0000","last_became_peered":"2024-09-22T03:18:23.103912+0000","last_unstale":"2024-09-22T03:19:01.317445+0000","last_undegraded":"2024-09-22T03:19:01.317445+0000","last_fullsized":"2024-09-22T03:19:01.317445+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:17:25.017919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:17:25.017919+0000","last_clean_scrub_stamp":"2024-09-22T03:17:25.017919+0000","objects_scrubbed":0,"log_size":575,"log_dups_size":0,"ondisk_log_size":575,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":576,"num_write_kb":4096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":2260992,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.5","version":"132'737","reported_seq":796,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-22T03:19:01.289276+0000","last_change":"2024-09-22T03:18:23.988488+0000","last_active":"2024-09-22T03:19:01.289276+0000","last_peered":"2024-09-22T03:19:01.289276+0000","last_clean":"2024-09-22T03:19:01.289276+0000","last_became_active":"2024-09-22T03:18:23.976152+0000","last_became_peered":"2024-09-22T03:18:23.976152+0000","last_unstale":"2024-09-22T03:19:01.289276+0000","last_undegraded":"2024-09-22T03:19:01.289276+0000","last_fullsized":"2024-09-22T03:19:01.289276+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:17:25.017919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:17:25.017919+0000","last_clean_scrub_stamp":"2024-09-22T03:17:25.017919+0000","objects_scrubbed":0,"log_size":737,"log_dups_size":0,"ondisk_log_size":737,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":738,"num_write_kb":5248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":2883584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.4","version":"132'782","reported_seq":957,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-22T03:19:01.374179+0000","last_change":"2024-09-22T03:18:52.169648+0000","last_active":"2024-09-22T03:19:01.374179+0000","last_peered":"2024-09-22T03:19:01.374179+0000","last_clean":"2024-09-22T03:19:01.374179+0000","last_became_active":"2024-09-22T03:18:52.054431+0000","last_became_peered":"2024-09-22T03:18:52.054431+0000","last_unstale":"2024-09-22T03:19:01.374179+0000","last_undegraded":"2024-09-22T03:19:01.374179+0000","last_fullsized":"2024-09-22T03:19:01.374179+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:17:25.017919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:17:25.017919+0000","last_clean_scrub_stamp":"2024-09-22T03:17:25.017919+0000","objects_scrubbed":0,"log_size":782,"log_dups_size":0,"ondisk_log_size":782,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":783,"num_write_kb":5568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":52,"num_bytes_recovered":3350528,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.3","version":"132'557","reported_seq":577,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-22T03:19:01.355660+0000","last_change":"2024-09-22T03:18:52.061628+0000","last_active":"2024-09-22T03:19:01.355660+0000","last_peered":"2024-09-22T03:19:01.355660+0000","last_clean":"2024-09-22T03:19:01.355660+0000","last_became_active":"2024-09-22T03:18:52.056643+0000","last_became_peered":"2024-09-22T03:18:52.056643+0000","last_unstale":"2024-09-22T03:19:01.355660+0000","last_undegraded":"2024-09-22T03:19:01.355660+0000","last_fullsized":"2024-09-22T03:19:01.355660+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:17:25.017919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:17:25.017919+0000","last_clean_scrub_stamp":"2024-09-22T03:17:25.017919+0000","objects_scrubbed":0,"log_size":557,"log_dups_size":0,"ondisk_log_size":557,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":558,"num_write_kb":3968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":2359296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.2","version":"132'692","reported_seq":769,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-22T03:19:01.351489+0000","last_change":"2024-09-22T03:18:52.037870+0000","last_active":"2024-09-22T03:19:01.351489+0000","last_peered":"2024-09-22T03:19:01.351489+0000","last_clean":"2024-09-22T03:19:01.351489+0000","last_became_active":"2024-09-22T03:18:52.018974+0000","last_became_peered":"2024-09-22T03:18:52.018974+0000","last_unstale":"2024-09-22T03:19:01.351489+0000","last_undegraded":"2024-09-22T03:19:01.351489+0000","last_fullsized":"2024-09-22T03:19:01.351489+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:17:25.017919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:17:25.017919+0000","last_clean_scrub_stamp":"2024-09-22T03:17:25.017919+0000","objects_scrubbed":0,"log_size":692,"log_dups_size":0,"ondisk_log_size":692,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":693,"num_write_kb":4928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":393216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1","version":"132'716","reported_seq":839,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-22T03:19:00.700113+0000","last_change":"2024-09-22T03:18:52.045135+0000","last_active":"2024-09-22T03:19:00.700113+0000","last_peered":"2024-09-22T03:19:00.700113+0000","last_clean":"2024-09-22T03:19:00.700113+0000","last_became_active":"2024-09-22T03:18:52.041328+0000","last_became_peered":"2024-09-22T03:18:52.041328+0000","last_unstale":"2024-09-22T03:19:00.700113+0000","last_undegraded":"2024-09-22T03:19:00.700113+0000","last_fullsized":"2024-09-22T03:19:00.700113+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:17:25.017919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:17:25.017919+0000","last_clean_scrub_stamp":"2024-09-22T03:17:25.017919+0000","objects_scrubbed":0,"log_size":716,"log_dups_size":0,"ondisk_log_size":716,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":786432,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":717,"num_write_kb":5184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":3342336,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"132'593","reported_seq":701,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-22T03:19:01.334938+0000","last_change":"2024-09-22T03:18:52.070377+0000","last_active":"2024-09-22T03:19:01.334938+0000","last_peered":"2024-09-22T03:19:01.334938+0000","last_clean":"2024-09-22T03:19:01.334938+0000","last_became_active":"2024-09-22T03:18:52.064222+0000","last_became_peered":"2024-09-22T03:18:52.064222+0000","last_unstale":"2024-09-22T03:19:01.334938+0000","last_undegraded":"2024-09-22T03:19:01.334938+0000","last_fullsized":"2024-09-22T03:19:01.334938+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:17:25.017919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:17:25.017919+0000","last_clean_scrub_stamp":"2024-09-22T03:17:25.017919+0000","objects_scrubbed":0,"log_size":593,"log_dups_size":0,"ondisk_log_size":593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":594,"num_write_kb":4224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":2490368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.9","version":"132'728","reported_seq":802,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-22T03:19:01.381715+0000","last_change":"2024-09-22T03:18:52.042972+0000","last_active":"2024-09-22T03:19:01.381715+0000","last_peered":"2024-09-22T03:19:01.381715+0000","last_clean":"2024-09-22T03:19:01.381715+0000","last_became_active":"2024-09-22T03:18:52.022349+0000","last_became_peered":"2024-09-22T03:18:52.022349+0000","last_unstale":"2024-09-22T03:19:01.381715+0000","last_undegraded":"2024-09-22T03:19:01.381715+0000","last_fullsized":"2024-09-22T03:19:01.381715+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:17:25.017919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:17:25.017919+0000","last_clean_scrub_stamp":"2024-09-22T03:17:25.017919+0000","objects_scrubbed":0,"log_size":728,"log_dups_size":0,"ondisk_log_size":728,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":729,"num_write_kb":5184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":393216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":164,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-22T03:18:53.942215+0000","last_change":"2024-09-22T03:18:53.943593+0000","last_active":"2024-09-22T03:18:53.942215+0000","last_peered":"2024-09-22T03:18:53.942215+0000","last_clean":"2024-09-22T03:18:53.942215+0000","last_became_active":"2024-09-22T03:18:52.032546+0000","last_became_peered":"2024-09-22T03:18:52.032546+0000","last_unstale":"2024-09-22T03:18:53.942215+0000","last_undegraded":"2024-09-22T03:18:53.942215+0000","last_fullsized":"2024-09-22T03:18:53.942215+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.8","version":"132'656","reported_seq":818,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-22T03:19:01.351441+0000","last_change":"2024-09-22T03:18:52.051422+0000","last_active":"2024-09-22T03:19:01.351441+0000","last_peered":"2024-09-22T03:19:01.351441+0000","last_clean":"2024-09-22T03:19:01.351441+0000","last_became_active":"2024-09-22T03:18:52.043925+0000","last_became_peered":"2024-09-22T03:18:52.043925+0000","last_unstale":"2024-09-22T03:19:01.351441+0000","last_undegraded":"2024-09-22T03:19:01.351441+0000","last_fullsized":"2024-09-22T03:19:01.351441+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:17:25.017919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:17:25.017919+0000","last_clean_scrub_stamp":"2024-09-22T03:17:25.017919+0000","objects_scrubbed":0,"log_size":656,"log_dups_size":0,"ondisk_log_size":656,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":657,"num_write_kb":4672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":2957312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":320,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-22T03:18:53.952624+0000","last_change":"2024-09-22T03:18:19.078674+0000","last_active":"2024-09-22T03:18:53.952624+0000","last_peered":"2024-09-22T03:18:53.952624+0000","last_clean":"2024-09-22T03:18:53.952624+0000","last_became_active":"2024-09-22T03:18:18.121868+0000","last_became_peered":"2024-09-22T03:18:18.121868+0000","last_unstale":"2024-09-22T03:18:53.952624+0000","last_undegraded":"2024-09-22T03:18:53.952624+0000","last_fullsized":"2024-09-22T03:18:53.952624+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.b","version":"132'530","reported_seq":656,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-22T03:19:01.320330+0000","last_change":"2024-09-22T03:18:23.986969+0000","last_active":"2024-09-22T03:19:01.320330+0000","last_peered":"2024-09-22T03:19:01.320330+0000","last_clean":"2024-09-22T03:19:01.320330+0000","last_became_active":"2024-09-22T03:18:23.981163+0000","last_became_peered":"2024-09-22T03:18:23.981163+0000","last_unstale":"2024-09-22T03:19:01.320330+0000","last_undegraded":"2024-09-22T03:19:01.320330+0000","last_fullsized":"2024-09-22T03:19:01.320330+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:17:25.017919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:17:25.017919+0000","last_clean_scrub_stamp":"2024-09-22T03:17:25.017919+0000","objects_scrubbed":0,"log_size":530,"log_dups_size":0,"ondisk_log_size":530,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":531,"num_write_kb":3776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":2031616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":319,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-22T03:18:53.941187+0000","last_change":"2024-09-22T03:18:53.942490+0000","last_active":"2024-09-22T03:18:53.941187+0000","last_peered":"2024-09-22T03:18:53.941187+0000","last_clean":"2024-09-22T03:18:53.941187+0000","last_became_active":"2024-09-22T03:18:51.995786+0000","last_became_peered":"2024-09-22T03:18:51.995786+0000","last_unstale":"2024-09-22T03:18:53.941187+0000","last_undegraded":"2024-09-22T03:18:53.941187+0000","last_fullsized":"2024-09-22T03:18:53.941187+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.a","version":"132'560","reported_seq":674,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-22T03:19:00.719307+0000","last_change":"2024-09-22T03:18:51.996503+0000","last_active":"2024-09-22T03:19:00.719307+0000","last_peered":"2024-09-22T03:19:00.719307+0000","last_clean":"2024-09-22T03:19:00.719307+0000","last_became_active":"2024-09-22T03:18:51.992601+0000","last_became_peered":"2024-09-22T03:18:51.992601+0000","last_unstale":"2024-09-22T03:19:00.719307+0000","last_undegraded":"2024-09-22T03:19:00.719307+0000","last_fullsized":"2024-09-22T03:19:00.719307+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:17:25.017919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:17:25.017919+0000","last_clean_scrub_stamp":"2024-09-22T03:17:25.017919+0000","objects_scrubbed":0,"log_size":560,"log_dups_size":0,"ondisk_log_size":560,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":983040,"num_objects":15,"num_object_clones":0,"num_object_copies":45,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":561,"num_write_kb":4096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39,"num_bytes_recovered":2408448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":192,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-22T03:18:53.941928+0000","last_change":"2024-09-22T03:18:53.943124+0000","last_active":"2024-09-22T03:18:53.941928+0000","last_peered":"2024-09-22T03:18:53.941928+0000","last_clean":"2024-09-22T03:18:53.941928+0000","last_became_active":"2024-09-22T03:18:52.037207+0000","last_became_peered":"2024-09-22T03:18:52.037207+0000","last_unstale":"2024-09-22T03:18:53.941928+0000","last_undegraded":"2024-09-22T03:18:53.941928+0000","last_fullsized":"2024-09-22T03:18:53.941928+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"12'97","reported_seq":441,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-22T03:18:53.951461+0000","last_change":"2024-09-22T03:18:18.126160+0000","last_active":"2024-09-22T03:18:53.951461+0000","last_peered":"2024-09-22T03:18:53.951461+0000","last_clean":"2024-09-22T03:18:53.951461+0000","last_became_active":"2024-09-22T03:18:18.122292+0000","last_became_peered":"2024-09-22T03:18:18.122292+0000","last_unstale":"2024-09-22T03:18:53.951461+0000","last_undegraded":"2024-09-22T03:18:53.951461+0000","last_fullsized":"2024-09-22T03:18:53.951461+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.d","version":"132'595","reported_seq":731,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-22T03:19:01.426510+0000","last_change":"2024-09-22T03:18:24.994136+0000","last_active":"2024-09-22T03:19:01.426510+0000","last_peered":"2024-09-22T03:19:01.426510+0000","last_clean":"2024-09-22T03:19:01.426510+0000","last_became_active":"2024-09-22T03:18:24.975245+0000","last_became_peered":"2024-09-22T03:18:24.975245+0000","last_unstale":"2024-09-22T03:19:01.426510+0000","last_undegraded":"2024-09-22T03:19:01.426510+0000","last_fullsized":"2024-09-22T03:19:01.426510+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:17:25.017919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:17:25.017919+0000","last_clean_scrub_stamp":"2024-09-22T03:17:25.017919+0000","objects_scrubbed":0,"log_size":595,"log_dups_size":0,"ondisk_log_size":595,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":596,"num_write_kb":4225,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":2359296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":270,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-22T03:18:53.947355+0000","last_change":"2024-09-22T03:18:19.044456+0000","last_active":"2024-09-22T03:18:53.947355+0000","last_peered":"2024-09-22T03:18:53.947355+0000","last_clean":"2024-09-22T03:18:53.947355+0000","last_became_active":"2024-09-22T03:18:18.129858+0000","last_became_peered":"2024-09-22T03:18:18.129858+0000","last_unstale":"2024-09-22T03:18:53.947355+0000","last_undegraded":"2024-09-22T03:18:53.947355+0000","last_fullsized":"2024-09-22T03:18:53.947355+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.c","version":"132'773","reported_seq":941,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-22T03:19:01.410577+0000","last_change":"2024-09-22T03:18:52.050097+0000","last_active":"2024-09-22T03:19:01.410577+0000","last_peered":"2024-09-22T03:19:01.410577+0000","last_clean":"2024-09-22T03:19:01.410577+0000","last_became_active":"2024-09-22T03:18:52.040757+0000","last_became_peered":"2024-09-22T03:18:52.040757+0000","last_unstale":"2024-09-22T03:19:01.410577+0000","last_undegraded":"2024-09-22T03:19:01.410577+0000","last_fullsized":"2024-09-22T03:19:01.410577+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:17:25.017919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:17:25.017919+0000","last_clean_scrub_stamp":"2024-09-22T03:17:25.017919+0000","objects_scrubbed":0,"log_size":773,"log_dups_size":0,"ondisk_log_size":773,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":774,"num_write_kb":5504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":51,"num_bytes_recovered":3342336,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":294,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-22T03:18:53.939802+0000","last_change":"2024-09-22T03:18:53.942094+0000","last_active":"2024-09-22T03:18:53.939802+0000","last_peered":"2024-09-22T03:18:53.939802+0000","last_clean":"2024-09-22T03:18:53.939802+0000","last_became_active":"2024-09-22T03:18:52.017091+0000","last_became_peered":"2024-09-22T03:18:52.017091+0000","last_unstale":"2024-09-22T03:18:53.939802+0000","last_undegraded":"2024-09-22T03:18:53.939802+0000","last_fullsized":"2024-09-22T03:18:53.939802+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.f","version":"132'682","reported_seq":820,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-22T03:19:00.657625+0000","last_change":"2024-09-22T03:18:52.023370+0000","last_active":"2024-09-22T03:19:00.657625+0000","last_peered":"2024-09-22T03:19:00.657625+0000","last_clean":"2024-09-22T03:19:00.657625+0000","last_became_active":"2024-09-22T03:18:52.017223+0000","last_became_peered":"2024-09-22T03:18:52.017223+0000","last_unstale":"2024-09-22T03:19:00.657625+0000","last_undegraded":"2024-09-22T03:19:00.657625+0000","last_fullsized":"2024-09-22T03:19:00.657625+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:17:25.017919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:17:25.017919+0000","last_clean_scrub_stamp":"2024-09-22T03:17:25.017919+0000","objects_scrubbed":0,"log_size":682,"log_dups_size":0,"ondisk_log_size":682,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":876544,"num_objects":14,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":683,"num_write_kb":4952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":3014656,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":320,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-22T03:18:53.942959+0000","last_change":"2024-09-22T03:18:53.946932+0000","last_active":"2024-09-22T03:18:53.942959+0000","last_peered":"2024-09-22T03:18:53.942959+0000","last_clean":"2024-09-22T03:18:53.942959+0000","last_became_active":"2024-09-22T03:18:52.015338+0000","last_became_peered":"2024-09-22T03:18:52.015338+0000","last_unstale":"2024-09-22T03:18:53.942959+0000","last_undegraded":"2024-09-22T03:18:53.942959+0000","last_fullsized":"2024-09-22T03:18:53.942959+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.e","version":"132'593","reported_seq":743,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-22T03:19:01.313020+0000","last_change":"2024-09-22T03:18:52.073922+0000","last_active":"2024-09-22T03:19:01.313020+0000","last_peered":"2024-09-22T03:19:01.313020+0000","last_clean":"2024-09-22T03:19:01.313020+0000","last_became_active":"2024-09-22T03:18:52.068257+0000","last_became_peered":"2024-09-22T03:18:52.068257+0000","last_unstale":"2024-09-22T03:19:01.313020+0000","last_undegraded":"2024-09-22T03:19:01.313020+0000","last_fullsized":"2024-09-22T03:19:01.313020+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:17:25.017919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:17:25.017919+0000","last_clean_scrub_stamp":"2024-09-22T03:17:25.017919+0000","objects_scrubbed":0,"log_size":593,"log_dups_size":0,"ondisk_log_size":593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":594,"num_write_kb":4224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":42,"num_bytes_recovered":2752512,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":313,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-22T03:18:53.942275+0000","last_change":"2024-09-22T03:18:53.947367+0000","last_active":"2024-09-22T03:18:53.942275+0000","last_peered":"2024-09-22T03:18:53.942275+0000","last_clean":"2024-09-22T03:18:53.942275+0000","last_became_active":"2024-09-22T03:18:52.023725+0000","last_became_peered":"2024-09-22T03:18:52.023725+0000","last_unstale":"2024-09-22T03:18:53.942275+0000","last_undegraded":"2024-09-22T03:18:53.942275+0000","last_fullsized":"2024-09-22T03:18:53.942275+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":2646016,"num_objects":41,"num_object_clones":0,"num_object_copies":123,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10558,"num_write_kb":75353,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":630,"num_bytes_recovered":40206336,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":6008832,"data_stored":12017664,"data_compressed":620541,"data_compressed_allocated":6008832,"data_compressed_original":12017664,"omap_allocated":0,"internal_metadata":0},"log_size":10542,"ondisk_log_size":10542,"up":48,"acting":48,"num_store_stats":4},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1180736,"data_compressed":59924,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":130,"seq":558345748489,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":55544,"kb_used_data":13800,"kb_used_omap":0,"kb_used_meta":41728,"kb_avail":94316296,"statfs":{"total":96636764160,"available":96579887104,"internally_reserved":0,"allocated":14131200,"data_stored":26488747,"data_compressed":1344717,"data_compressed_allocated":13021184,"data_compressed_original":26042368,"omap_allocated":0,"internal_metadata":42729472},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":100,"seq":429496729764,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":36152,"kb_used_data":3560,"kb_used_omap":0,"kb_used_meta":32576,"kb_avail":94335688,"statfs":{"total":96636764160,"available":96599744512,"internally_reserved":0,"allocated":3645440,"data_stored":5976547,"data_compressed":285031,"data_compressed_allocated":2510848,"data_compressed_original":5513216,"omap_allocated":0,"internal_metadata":33357824},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":121,"seq":519691042861,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":38664,"kb_used_data":1144,"kb_used_omap":0,"kb_used_meta":37504,"kb_avail":94333176,"statfs":{"total":96636764160,"available":96597172224,"internally_reserved":0,"allocated":1171456,"data_stored":1036747,"data_compressed":29962,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":38404096},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":73,"seq":313532612873,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":51784,"kb_used_data":1592,"kb_used_omap":0,"kb_used_meta":50176,"kb_avail":94320056,"statfs":{"total":96636764160,"available":96583737344,"internally_reserved":0,"allocated":1630208,"data_stored":1954256,"data_compressed":77338,"data_compressed_allocated":499712,"data_compressed_original":1490944,"omap_allocated":0,"internal_metadata":51380224},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29962,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29962,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":6008832,"data_stored":12017664,"data_compressed":620541,"data_compressed_allocated":6008832,"data_compressed_original":12017664,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-22T03:19:03.573 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-22T03:19:03.683 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:19:03.680+0000 7f5649f57640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:19:03.687 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:19:03.685+0000 7f5649f57640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:19:04.670 INFO:tasks.ceph.ceph_manager.ceph:removing pool_name unique_pool_5 2024-09-22T03:19:04.671 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm unique_pool_5 unique_pool_5 --yes-i-really-really-mean-it 2024-09-22T03:19:04.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:19:04.771+0000 7fb4c014e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:19:04.783 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:19:04.782+0000 7fb4c014e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:19:04.974 INFO:teuthology.orchestra.run.smithi153.stderr:pool 'unique_pool_5' removed 2024-09-22T03:19:04.991 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-09-22T03:19:04.991 INFO:tasks.radosbench:Beginning radosbench... 2024-09-22T03:19:04.992 INFO:tasks.ceph.ceph_manager.ceph:creating pool_name unique_pool_6 2024-09-22T03:19:04.992 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create unique_pool_6 16 2024-09-22T03:19:05.133 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:19:05.123+0000 7f0a91997640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:19:05.137 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:19:05.136+0000 7f0a91997640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:19:05.757 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-22T03:19:05.859 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:19:05.856+0000 7fca804b2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:19:05.862 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:19:05.861+0000 7fca804b2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:19:05.978 INFO:teuthology.orchestra.run.smithi153.stderr:pool 'unique_pool_6' created 2024-09-22T03:19:05.996 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool application enable unique_pool_6 rados --yes-i-really-mean-it || true 2024-09-22T03:19:06.117 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:19:06.106+0000 7f6416797640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:19:06.120 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:19:06.118+0000 7f6416797640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:19:06.981 INFO:teuthology.orchestra.run.smithi153.stderr:enabled application 'rados' on pool 'unique_pool_6' 2024-09-22T03:19:07.665 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 2, 1, 3] 2024-09-22T03:19:07.665 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-22T03:19:07.665 INFO:tasks.thrashosds.thrasher:Killing osd 1, live_osds are [0, 2, 1, 3] 2024-09-22T03:19:07.665 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-09-22T03:19:07.666 INFO:teuthology.orchestra.run:waiting for 300 2024-09-22T03:19:07.930 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-22T03:19:08.008 DEBUG:teuthology.orchestra.run.smithi153:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 8192 --object-size 65536 -p unique_pool_6 bench 90 write' 2024-09-22T03:19:08.010 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-09-22T03:19:08.010 INFO:teuthology.orchestra.run:waiting for 3000 2024-09-22T03:19:08.041 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:19:08.037+0000 7f4c15ad7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:19:08.044 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:19:08.042+0000 7f4c15ad7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:19:08.066 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:hints = 1 2024-09-22T03:19:08.066 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:Maintaining 16 concurrent writes of 8192 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-09-22T03:19:08.067 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:Object prefix: benchmark_data_smithi153_43901 2024-09-22T03:19:08.067 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-22T03:19:08.067 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 0 0 0 0 0 0 - 0 2024-09-22T03:19:09.067 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 1 16 16 0 0 0 - 0 2024-09-22T03:19:10.067 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 2 16 16 0 0 0 - 0 2024-09-22T03:19:11.067 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 3 16 16 0 0 0 - 0 2024-09-22T03:19:11.719 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~0s 2024-09-22T03:19:12.067 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 4 16 16 0 0 0 - 0 2024-09-22T03:19:13.067 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 5 16 16 0 0 0 - 0 2024-09-22T03:19:13.766 INFO:tasks.ceph.osd.1:Stopped 2024-09-22T03:19:14.067 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 6 16 16 0 0 0 - 0 2024-09-22T03:19:15.067 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 7 16 16 0 0 0 - 0 2024-09-22T03:19:16.067 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 8 16 16 0 0 0 - 0 2024-09-22T03:19:17.068 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 9 16 16 0 0 0 - 0 2024-09-22T03:19:17.223 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 has been restored 2024-09-22T03:19:18.068 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 10 16 16 0 0 0 - 0 2024-09-22T03:19:18.767 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [1] live_osds: [0, 2, 3] 2024-09-22T03:19:18.768 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-22T03:19:18.768 INFO:tasks.thrashosds.thrasher:Reviving osd 1 2024-09-22T03:19:18.768 INFO:tasks.ceph.osd.1:Restarting daemon 2024-09-22T03:19:18.768 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2>> /var/log/ceph/ceph-osd.1.log 2024-09-22T03:19:18.771 INFO:tasks.ceph.osd.1:Started 2024-09-22T03:19:18.771 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-22T03:19:18.881 INFO:teuthology.orchestra.run.smithi153.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-09-22T03:19:18.885 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-22T03:19:18.885 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2024-09-22T03:19:19.068 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 11 16 16 0 0 0 - 0 2024-09-22T03:19:20.068 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 12 16 16 0 0 0 - 0 2024-09-22T03:19:21.068 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 13 16 16 0 0 0 - 0 2024-09-22T03:19:22.068 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 14 16 16 0 0 0 - 0 2024-09-22T03:19:23.068 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 15 16 16 0 0 0 - 0 2024-09-22T03:19:23.886 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-22T03:19:23.989 INFO:teuthology.orchestra.run.smithi153.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-09-22T03:19:23.991 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-22T03:19:23.992 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2024-09-22T03:19:24.068 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 16 16 16 0 0 0 - 0 2024-09-22T03:19:25.068 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 17 16 16 0 0 0 - 0 2024-09-22T03:19:26.068 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 18 16 16 0 0 0 - 0 2024-09-22T03:19:26.947 INFO:teuthology.orchestra.run.smithi153.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-09-22T03:19:27.036 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-22T03:19:27.036 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-22T03:19:27.069 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 19 16 16 0 0 0 - 0 2024-09-22T03:19:28.069 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:2024-09-22T03:19:28.068296+0000 min lat: 9999 max lat: 0 avg lat: 0 2024-09-22T03:19:28.069 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-22T03:19:28.069 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 20 16 16 0 0 0 - 0 2024-09-22T03:19:28.993 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-22T03:19:29.038 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-22T03:19:29.069 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 21 16 16 0 0 0 - 0 2024-09-22T03:19:29.163 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-22T03:19:29.163 INFO:teuthology.orchestra.run.smithi153.stdout: "ops_in_flight": [ 2024-09-22T03:19:29.163 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-22T03:19:29.163 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "replicated_request", 2024-09-22T03:19:29.163 INFO:teuthology.orchestra.run.smithi153.stdout: "id": 177, 2024-09-22T03:19:29.163 INFO:teuthology.orchestra.run.smithi153.stdout: "detail": { 2024-09-22T03:19:29.164 INFO:teuthology.orchestra.run.smithi153.stdout: "RepRequest": { 2024-09-22T03:19:29.164 INFO:teuthology.orchestra.run.smithi153.stdout: "reqid": "client.5101.0:4", 2024-09-22T03:19:29.164 INFO:teuthology.orchestra.run.smithi153.stdout: "pgid": "9.1", 2024-09-22T03:19:29.164 INFO:teuthology.orchestra.run.smithi153.stdout: "map_epoch": 139, 2024-09-22T03:19:29.164 INFO:teuthology.orchestra.run.smithi153.stdout: "min_epoch": 138, 2024-09-22T03:19:29.164 INFO:teuthology.orchestra.run.smithi153.stdout: "oid": "9:8f286ba0:::benchmark_data_smithi153_43901_object0:head", 2024-09-22T03:19:29.164 INFO:teuthology.orchestra.run.smithi153.stdout: "from": "0" 2024-09-22T03:19:29.164 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:19:29.164 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:19:29.164 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.164 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-22T03:19:29.164 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "client_request", 2024-09-22T03:19:29.164 INFO:teuthology.orchestra.run.smithi153.stdout: "id": 57, 2024-09-22T03:19:29.165 INFO:teuthology.orchestra.run.smithi153.stdout: "detail": { 2024-09-22T03:19:29.165 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:19:29.165 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:19:29.165 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:19:28.009338+0000" 2024-09-22T03:19:29.165 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.165 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:19:29.165 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:19:29.165 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:19:28.009345+0000" 2024-09-22T03:19:29.165 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.165 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:19:29.165 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:19:29.165 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.166 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.166 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:19:29.166 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:19:29.166 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:19:28.009615+0000" 2024-09-22T03:19:29.166 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.166 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:19:29.166 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:19:29.166 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.166 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.166 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:19:29.166 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:19:29.166 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.166 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.166 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:19:29.167 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:19:29.167 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.167 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.167 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:19:29.167 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:19:29.167 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:19:28.009749+0000" 2024-09-22T03:19:29.167 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.167 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:19:29.167 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:19:29.167 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.167 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.167 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:19:29.168 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:19:29.168 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:19:28.010320+0000" 2024-09-22T03:19:29.168 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.168 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:19:29.168 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:19:29.168 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.168 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.168 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:19:29.168 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:19:29.168 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.168 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.168 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:19:29.168 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:19:29.168 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.169 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.169 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:19:29.169 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:19:29.169 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.169 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.169 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:19:29.169 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:19:29.169 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:19:28.010720+0000" 2024-09-22T03:19:29.169 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.169 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:19:29.169 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:19:29.169 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.169 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.170 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:19:29.170 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:19:29.170 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.170 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.170 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:19:29.170 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:19:29.170 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.170 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.170 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:19:29.170 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:19:29.170 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:19:28.010943+0000" 2024-09-22T03:19:29.171 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.171 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:19:29.171 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:19:29.171 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.171 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.171 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:19:29.171 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:19:29.171 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.171 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.171 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:19:29.171 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:19:29.171 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.171 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.171 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:19:29.172 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:19:29.172 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:19:29.060777+0000" 2024-09-22T03:19:29.172 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.172 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:19:29.172 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:19:29.172 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.172 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.172 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:19:29.172 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:19:29.172 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.172 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.172 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:19:29.172 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:19:29.173 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.173 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.173 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:19:29.173 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:19:29.173 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.173 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.173 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:19:29.173 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:19:29.173 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.173 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.173 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:19:29.173 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:19:29.173 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.174 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.174 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:19:29.174 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:19:29.174 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.174 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.174 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:19:29.174 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:19:29.174 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:19:29.061951+0000" 2024-09-22T03:19:29.174 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.174 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:19:29.174 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:19:29.174 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.174 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.175 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:19:29.175 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:19:29.175 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.175 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.175 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:19:29.175 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:19:29.175 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.175 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.175 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:19:29.175 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:19:29.175 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:19:29.120519+0000" 2024-09-22T03:19:29.175 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.175 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:19:29.175 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:19:29.176 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.176 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.176 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:19:29.176 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:19:29.176 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.176 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.176 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:19:29.176 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:19:29.176 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.176 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.176 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:19:29.177 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:19:29.177 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.177 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.177 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:19:29.177 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:19:29.177 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.177 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.177 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:19:29.177 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:19:29.177 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.177 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:19:29.177 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:19:29.177 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.177 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-22T03:19:29.178 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "client_request", 2024-09-22T03:19:29.178 INFO:teuthology.orchestra.run.smithi153.stdout: "id": 61, 2024-09-22T03:19:29.178 INFO:teuthology.orchestra.run.smithi153.stdout: "detail": { 2024-09-22T03:19:29.178 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:19:29.178 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:19:29.178 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:19:28.011304+0000" 2024-09-22T03:19:29.178 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.178 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:19:29.178 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:19:29.178 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:19:28.011311+0000" 2024-09-22T03:19:29.178 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.178 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:19:29.178 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:19:29.179 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.179 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.179 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:19:29.179 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:19:29.179 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:19:28.011614+0000" 2024-09-22T03:19:29.179 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.179 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:19:29.179 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:19:29.179 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.179 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.179 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:19:29.180 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:19:29.180 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.180 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.180 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:19:29.180 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:19:29.180 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.180 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.180 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:19:29.180 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:19:29.180 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:19:28.011786+0000" 2024-09-22T03:19:29.180 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.180 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:19:29.180 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:19:29.180 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.180 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.181 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:19:29.181 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:19:29.181 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:19:28.013948+0000" 2024-09-22T03:19:29.181 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.181 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:19:29.181 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:19:29.181 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.181 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.181 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:19:29.181 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:19:29.181 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.181 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.181 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:19:29.182 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:19:29.182 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.182 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.182 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:19:29.182 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:19:29.182 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.182 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.182 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:19:29.182 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:19:29.182 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:19:28.014384+0000" 2024-09-22T03:19:29.182 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.182 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:19:29.182 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:19:29.182 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.183 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.183 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:19:29.183 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:19:29.183 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.183 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.183 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:19:29.183 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:19:29.183 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.183 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.183 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:19:29.183 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:19:29.183 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:19:29.060905+0000" 2024-09-22T03:19:29.183 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.183 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:19:29.183 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:19:29.184 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.184 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.184 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:19:29.184 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:19:29.184 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.184 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.184 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:19:29.184 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:19:29.184 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.184 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.184 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:19:29.184 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:19:29.184 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:19:29.061697+0000" 2024-09-22T03:19:29.184 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.185 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:19:29.185 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:19:29.185 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.185 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.185 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:19:29.185 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:19:29.185 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.185 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.185 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:19:29.186 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:19:29.186 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.186 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.186 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:19:29.186 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:19:29.186 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.186 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.186 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:19:29.186 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:19:29.186 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.187 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.187 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:19:29.187 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:19:29.187 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.187 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.187 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:19:29.187 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:19:29.187 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.187 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.187 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:19:29.188 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:19:29.188 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:19:29.063175+0000" 2024-09-22T03:19:29.188 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.188 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:19:29.188 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:19:29.188 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.188 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.188 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:19:29.188 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:19:29.188 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.188 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.188 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:19:29.189 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:19:29.189 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.189 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.189 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:19:29.189 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:19:29.189 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:19:29.121662+0000" 2024-09-22T03:19:29.189 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.189 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:19:29.189 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:19:29.189 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.189 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.189 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:19:29.189 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:19:29.189 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.190 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.190 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:19:29.190 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:19:29.190 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.190 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.190 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:19:29.190 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:19:29.190 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.190 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.191 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:19:29.191 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:19:29.191 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.191 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.191 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:19:29.191 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:19:29.191 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.191 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:19:29.191 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:19:29.191 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.191 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-22T03:19:29.191 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "client_request", 2024-09-22T03:19:29.191 INFO:teuthology.orchestra.run.smithi153.stdout: "id": 65, 2024-09-22T03:19:29.191 INFO:teuthology.orchestra.run.smithi153.stdout: "detail": { 2024-09-22T03:19:29.192 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:19:29.192 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:19:29.192 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:19:28.013728+0000" 2024-09-22T03:19:29.192 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.192 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:19:29.192 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:19:29.192 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:19:28.013736+0000" 2024-09-22T03:19:29.192 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.192 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:19:29.192 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:19:29.192 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.192 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.192 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:19:29.192 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:19:29.193 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:19:28.013858+0000" 2024-09-22T03:19:29.193 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.193 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:19:29.193 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:19:29.193 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.193 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.193 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:19:29.193 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:19:29.193 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.193 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.193 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:19:29.193 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:19:29.193 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.193 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.194 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:19:29.194 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:19:29.194 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:19:28.014095+0000" 2024-09-22T03:19:29.194 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.194 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:19:29.194 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:19:29.194 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.194 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.194 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:19:29.194 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:19:29.194 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:19:28.014647+0000" 2024-09-22T03:19:29.194 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.194 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:19:29.194 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:19:29.195 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.195 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.195 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:19:29.195 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:19:29.195 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.195 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.195 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:19:29.195 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:19:29.195 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.195 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.195 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:19:29.195 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:19:29.195 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.195 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.196 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:19:29.196 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:19:29.196 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:19:29.061266+0000" 2024-09-22T03:19:29.196 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.196 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:19:29.196 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:19:29.196 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.196 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.196 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:19:29.196 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:19:29.196 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.196 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.196 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:19:29.196 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:19:29.197 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.197 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.197 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:19:29.197 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:19:29.197 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:19:29.062130+0000" 2024-09-22T03:19:29.197 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.197 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:19:29.197 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:19:29.197 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.197 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.197 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:19:29.197 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:19:29.197 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.197 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.198 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:19:29.198 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:19:29.198 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.198 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.198 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:19:29.198 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:19:29.198 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:19:29.065848+0000" 2024-09-22T03:19:29.198 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.198 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:19:29.198 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:19:29.199 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.199 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.199 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:19:29.199 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:19:29.199 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.199 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.199 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:19:29.199 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:19:29.199 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.199 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.199 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:19:29.199 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:19:29.199 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.199 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.200 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:19:29.200 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:19:29.200 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.200 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.200 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:19:29.200 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:19:29.200 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.200 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.200 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:19:29.200 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:19:29.200 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.200 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.200 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:19:29.200 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:19:29.201 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:19:29.066643+0000" 2024-09-22T03:19:29.201 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.201 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:19:29.201 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:19:29.201 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.201 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.201 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:19:29.201 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:19:29.201 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.201 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.201 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:19:29.201 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:19:29.202 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:19:29.121766+0000" 2024-09-22T03:19:29.202 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.202 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:19:29.202 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:19:29.202 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:19:29.125624+0000" 2024-09-22T03:19:29.202 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.202 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:19:29.202 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:19:29.202 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.202 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.202 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:19:29.202 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:19:29.202 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.202 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.203 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:19:29.203 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:19:29.203 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.203 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.203 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:19:29.203 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:19:29.203 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.203 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.203 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:19:29.203 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:19:29.203 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.204 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.204 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:19:29.204 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:19:29.204 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.204 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:19:29.204 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:19:29.204 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.204 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-22T03:19:29.204 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "client_request", 2024-09-22T03:19:29.204 INFO:teuthology.orchestra.run.smithi153.stdout: "id": 68, 2024-09-22T03:19:29.204 INFO:teuthology.orchestra.run.smithi153.stdout: "detail": { 2024-09-22T03:19:29.204 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:19:29.204 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:19:29.204 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:19:28.014956+0000" 2024-09-22T03:19:29.205 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.205 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:19:29.205 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:19:29.205 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:19:28.014962+0000" 2024-09-22T03:19:29.205 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.205 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:19:29.205 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:19:29.205 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.205 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.205 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:19:29.205 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:19:29.205 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:19:28.015328+0000" 2024-09-22T03:19:29.205 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.206 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:19:29.206 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:19:29.206 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.206 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.206 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:19:29.206 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:19:29.206 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.206 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.206 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:19:29.206 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:19:29.206 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.206 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.206 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:19:29.206 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:19:29.207 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:19:28.016268+0000" 2024-09-22T03:19:29.207 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.207 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:19:29.207 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:19:29.207 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.207 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.207 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:19:29.207 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:19:29.207 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:19:28.016899+0000" 2024-09-22T03:19:29.207 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.208 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:19:29.208 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:19:29.208 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.208 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.208 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:19:29.208 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:19:29.208 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.208 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.208 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:19:29.208 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:19:29.208 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.208 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.208 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:19:29.208 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:19:29.209 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.209 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.209 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:19:29.209 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:19:29.209 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:19:29.062869+0000" 2024-09-22T03:19:29.209 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.209 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:19:29.209 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:19:29.209 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.209 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.209 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:19:29.209 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:19:29.209 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.210 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.210 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:19:29.210 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:19:29.210 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.210 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.210 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:19:29.210 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:19:29.210 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:19:29.065737+0000" 2024-09-22T03:19:29.210 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.210 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:19:29.210 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:19:29.210 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.210 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.210 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:19:29.210 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:19:29.211 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.211 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.211 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:19:29.211 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:19:29.211 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.211 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.211 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:19:29.211 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:19:29.211 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:19:29.069781+0000" 2024-09-22T03:19:29.211 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.211 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:19:29.211 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:19:29.211 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.211 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.211 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:19:29.212 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:19:29.212 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.212 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.212 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:19:29.212 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:19:29.212 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.212 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.212 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:19:29.212 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:19:29.212 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.212 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.212 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:19:29.212 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:19:29.213 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.213 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.213 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:19:29.213 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:19:29.213 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.213 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.213 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:19:29.213 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:19:29.213 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.213 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.213 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:19:29.213 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:19:29.213 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:19:29.071028+0000" 2024-09-22T03:19:29.213 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.213 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:19:29.214 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:19:29.214 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.214 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.214 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:19:29.214 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:19:29.214 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.214 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.214 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:19:29.214 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:19:29.214 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:19:29.120859+0000" 2024-09-22T03:19:29.214 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.214 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:19:29.214 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:19:29.214 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:19:29.127853+0000" 2024-09-22T03:19:29.215 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.215 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:19:29.215 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:19:29.215 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.215 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.215 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:19:29.215 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:19:29.215 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.215 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.215 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:19:29.215 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:19:29.215 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.215 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.215 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:19:29.215 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:19:29.216 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.216 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.216 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:19:29.216 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:19:29.216 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.216 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.216 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:19:29.216 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:19:29.216 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.217 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:19:29.217 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:19:29.217 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.217 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-22T03:19:29.217 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "client_request", 2024-09-22T03:19:29.217 INFO:teuthology.orchestra.run.smithi153.stdout: "id": 70, 2024-09-22T03:19:29.217 INFO:teuthology.orchestra.run.smithi153.stdout: "detail": { 2024-09-22T03:19:29.217 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:19:29.217 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:19:29.217 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:19:28.017072+0000" 2024-09-22T03:19:29.217 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.217 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:19:29.217 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:19:29.217 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:19:28.017078+0000" 2024-09-22T03:19:29.218 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.218 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:19:29.218 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:19:29.218 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.218 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.218 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:19:29.218 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:19:29.218 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:19:28.017194+0000" 2024-09-22T03:19:29.218 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.218 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:19:29.218 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:19:29.218 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.218 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.219 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:19:29.219 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:19:29.219 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.219 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.219 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:19:29.219 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:19:29.219 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.219 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.219 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:19:29.219 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:19:29.219 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:19:28.017539+0000" 2024-09-22T03:19:29.219 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.219 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:19:29.220 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:19:29.220 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.220 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.220 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:19:29.220 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:19:29.220 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:19:28.018209+0000" 2024-09-22T03:19:29.220 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.220 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:19:29.220 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:19:29.220 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.220 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.220 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:19:29.220 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:19:29.220 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.220 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.221 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:19:29.221 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:19:29.221 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.221 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.221 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:19:29.221 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:19:29.221 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.221 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.221 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:19:29.221 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:19:29.221 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:19:29.066272+0000" 2024-09-22T03:19:29.221 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.221 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:19:29.222 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:19:29.222 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.222 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.222 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:19:29.222 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:19:29.222 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.222 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.222 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:19:29.222 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:19:29.222 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.222 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.222 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:19:29.222 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:19:29.222 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:19:29.069649+0000" 2024-09-22T03:19:29.223 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.223 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:19:29.223 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:19:29.223 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.223 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.223 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:19:29.223 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:19:29.223 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.223 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.223 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:19:29.223 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:19:29.223 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.223 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.223 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:19:29.224 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:19:29.224 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:19:29.071224+0000" 2024-09-22T03:19:29.224 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.224 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:19:29.224 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:19:29.224 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.224 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.224 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:19:29.224 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:19:29.224 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.224 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.224 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:19:29.224 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:19:29.225 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.225 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.225 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:19:29.225 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:19:29.225 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.225 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.225 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:19:29.225 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:19:29.225 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.225 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.225 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:19:29.225 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:19:29.225 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.226 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.226 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:19:29.226 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:19:29.226 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.226 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.226 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:19:29.226 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:19:29.226 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:19:29.074068+0000" 2024-09-22T03:19:29.226 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.226 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:19:29.226 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:19:29.226 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.226 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.226 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:19:29.227 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:19:29.227 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.227 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.227 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:19:29.227 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:19:29.227 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:19:29.121405+0000" 2024-09-22T03:19:29.227 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.227 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:19:29.227 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:19:29.227 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:19:29.130035+0000" 2024-09-22T03:19:29.227 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.227 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:19:29.227 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:19:29.228 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.228 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.228 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:19:29.228 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:19:29.228 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.228 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.228 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:19:29.228 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:19:29.228 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.228 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.228 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:19:29.228 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:19:29.228 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.229 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.229 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:19:29.229 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:19:29.229 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.229 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.229 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:19:29.229 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:19:29.229 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.229 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:19:29.229 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:19:29.229 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.229 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-22T03:19:29.229 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "client_request", 2024-09-22T03:19:29.229 INFO:teuthology.orchestra.run.smithi153.stdout: "id": 73, 2024-09-22T03:19:29.230 INFO:teuthology.orchestra.run.smithi153.stdout: "detail": { 2024-09-22T03:19:29.230 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:19:29.230 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:19:29.230 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:19:28.018721+0000" 2024-09-22T03:19:29.230 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.230 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:19:29.230 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:19:29.230 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:19:28.018727+0000" 2024-09-22T03:19:29.230 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.230 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:19:29.230 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:19:29.230 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.231 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.231 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:19:29.231 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:19:29.231 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:19:28.018847+0000" 2024-09-22T03:19:29.231 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.231 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:19:29.231 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:19:29.231 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.231 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.231 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:19:29.231 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:19:29.231 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.231 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.231 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:19:29.232 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:19:29.232 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.232 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.232 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:19:29.232 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:19:29.232 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:19:28.018952+0000" 2024-09-22T03:19:29.232 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.232 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:19:29.232 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:19:29.232 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.232 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.232 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:19:29.233 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:19:29.233 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:19:28.020503+0000" 2024-09-22T03:19:29.233 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.233 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:19:29.233 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:19:29.233 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.233 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.233 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:19:29.233 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:19:29.233 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.233 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.233 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:19:29.234 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:19:29.234 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.234 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.234 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:19:29.234 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:19:29.234 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.234 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.234 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:19:29.234 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:19:29.234 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:19:29.070514+0000" 2024-09-22T03:19:29.234 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.235 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:19:29.235 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:19:29.235 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.235 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.235 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:19:29.235 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:19:29.235 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.235 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.235 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:19:29.235 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:19:29.235 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.235 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.236 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:19:29.236 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:19:29.236 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:19:29.071907+0000" 2024-09-22T03:19:29.236 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.236 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:19:29.236 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:19:29.236 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.236 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.236 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:19:29.236 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:19:29.236 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.236 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.236 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:19:29.236 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:19:29.237 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.237 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.237 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:19:29.237 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:19:29.237 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:19:29.074584+0000" 2024-09-22T03:19:29.237 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.237 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:19:29.237 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:19:29.237 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.237 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.237 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:19:29.237 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:19:29.237 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.238 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.238 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:19:29.238 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:19:29.238 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.238 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.238 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:19:29.238 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:19:29.238 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.238 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.238 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:19:29.238 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:19:29.238 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.238 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.238 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:19:29.239 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:19:29.239 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.239 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.239 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:19:29.239 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:19:29.239 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.239 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.239 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:19:29.239 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:19:29.239 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:19:29.076435+0000" 2024-09-22T03:19:29.239 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.239 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:19:29.239 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:19:29.239 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.240 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.240 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:19:29.240 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:19:29.240 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.240 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.240 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:19:29.240 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:19:29.240 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:19:29.120759+0000" 2024-09-22T03:19:29.240 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.240 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:19:29.240 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:19:29.240 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:19:29.133068+0000", 2024-09-22T03:19:29.240 INFO:teuthology.orchestra.run.smithi153.stdout: "blocker": { 2024-09-22T03:19:29.240 INFO:teuthology.orchestra.run.smithi153.stdout: "op_type": "CommonPGPipeline::process", 2024-09-22T03:19:29.240 INFO:teuthology.orchestra.run.smithi153.stdout: "detail": { 2024-09-22T03:19:29.241 INFO:teuthology.orchestra.run.smithi153.stdout: "waiting": 1, 2024-09-22T03:19:29.241 INFO:teuthology.orchestra.run.smithi153.stdout: "held_by_operation_id": 70 2024-09-22T03:19:29.241 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:19:29.241 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:19:29.241 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.241 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:19:29.241 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:19:29.241 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.241 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.241 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:19:29.241 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:19:29.242 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.242 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.242 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:19:29.242 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:19:29.242 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.242 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.242 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:19:29.242 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:19:29.242 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.242 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.242 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:19:29.242 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:19:29.242 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.243 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.243 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:19:29.243 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:19:29.243 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.243 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:19:29.243 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:19:29.243 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.243 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-22T03:19:29.243 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "client_request", 2024-09-22T03:19:29.243 INFO:teuthology.orchestra.run.smithi153.stdout: "id": 75, 2024-09-22T03:19:29.243 INFO:teuthology.orchestra.run.smithi153.stdout: "detail": { 2024-09-22T03:19:29.243 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:19:29.243 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:19:29.243 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:19:28.020238+0000" 2024-09-22T03:19:29.244 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.244 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:19:29.244 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:19:29.244 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:19:28.020245+0000" 2024-09-22T03:19:29.244 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.244 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:19:29.244 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:19:29.244 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.244 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.244 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:19:29.244 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:19:29.244 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:19:28.020331+0000" 2024-09-22T03:19:29.245 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.245 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:19:29.245 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:19:29.245 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.245 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.245 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:19:29.245 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:19:29.245 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.245 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.245 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:19:29.245 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:19:29.245 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.245 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.245 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:19:29.246 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:19:29.246 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:19:28.020448+0000" 2024-09-22T03:19:29.246 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.246 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:19:29.246 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:19:29.246 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.246 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.246 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:19:29.246 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:19:29.246 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:19:28.020883+0000" 2024-09-22T03:19:29.246 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.246 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:19:29.247 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:19:29.247 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.247 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.247 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:19:29.247 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:19:29.247 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.247 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.247 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:19:29.247 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:19:29.247 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.247 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.247 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:19:29.247 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:19:29.247 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.248 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.248 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:19:29.248 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:19:29.248 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:19:29.072837+0000" 2024-09-22T03:19:29.248 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.248 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:19:29.248 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:19:29.248 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.248 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.248 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:19:29.248 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:19:29.248 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.248 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.248 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:19:29.249 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:19:29.249 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.249 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.249 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:19:29.249 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:19:29.249 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:19:29.074999+0000" 2024-09-22T03:19:29.249 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.249 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:19:29.249 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:19:29.249 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.249 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.249 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:19:29.249 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:19:29.249 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.250 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.250 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:19:29.250 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:19:29.250 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.250 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.250 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:19:29.250 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:19:29.250 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:19:29.077299+0000" 2024-09-22T03:19:29.250 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.250 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:19:29.250 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:19:29.250 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.250 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.250 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:19:29.251 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:19:29.251 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.251 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.251 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:19:29.251 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:19:29.251 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.251 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.251 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:19:29.251 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:19:29.251 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.251 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.251 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:19:29.251 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:19:29.252 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.252 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.252 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:19:29.252 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:19:29.252 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.252 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.252 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:19:29.252 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:19:29.252 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.252 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.252 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:19:29.252 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:19:29.252 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:19:29.091281+0000" 2024-09-22T03:19:29.252 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.253 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:19:29.253 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:19:29.253 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.253 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.253 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:19:29.253 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:19:29.253 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.253 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.253 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:19:29.253 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:19:29.253 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:19:29.121037+0000" 2024-09-22T03:19:29.253 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.253 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:19:29.253 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:19:29.254 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.254 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.254 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:19:29.254 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:19:29.254 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.254 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.254 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:19:29.254 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:19:29.254 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.254 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.254 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:19:29.254 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:19:29.254 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.254 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.255 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:19:29.255 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:19:29.255 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.255 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.255 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:19:29.255 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:19:29.255 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.255 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.255 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:19:29.255 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:19:29.255 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.255 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:19:29.255 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:19:29.255 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.256 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-22T03:19:29.256 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "replicated_request", 2024-09-22T03:19:29.256 INFO:teuthology.orchestra.run.smithi153.stdout: "id": 175, 2024-09-22T03:19:29.256 INFO:teuthology.orchestra.run.smithi153.stdout: "detail": { 2024-09-22T03:19:29.256 INFO:teuthology.orchestra.run.smithi153.stdout: "RepRequest": { 2024-09-22T03:19:29.256 INFO:teuthology.orchestra.run.smithi153.stdout: "reqid": "client.5101.0:17", 2024-09-22T03:19:29.256 INFO:teuthology.orchestra.run.smithi153.stdout: "pgid": "9.0", 2024-09-22T03:19:29.256 INFO:teuthology.orchestra.run.smithi153.stdout: "map_epoch": 139, 2024-09-22T03:19:29.256 INFO:teuthology.orchestra.run.smithi153.stdout: "min_epoch": 138, 2024-09-22T03:19:29.256 INFO:teuthology.orchestra.run.smithi153.stdout: "oid": "9:034c39e8:::benchmark_data_smithi153_43901_object2:head", 2024-09-22T03:19:29.256 INFO:teuthology.orchestra.run.smithi153.stdout: "from": "2" 2024-09-22T03:19:29.256 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:19:29.256 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:19:29.257 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.257 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-22T03:19:29.257 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "client_request", 2024-09-22T03:19:29.257 INFO:teuthology.orchestra.run.smithi153.stdout: "id": 53, 2024-09-22T03:19:29.257 INFO:teuthology.orchestra.run.smithi153.stdout: "detail": { 2024-09-22T03:19:29.257 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:19:29.257 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:19:29.257 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:19:28.006536+0000" 2024-09-22T03:19:29.257 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.257 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:19:29.257 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:19:29.257 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:19:28.006555+0000" 2024-09-22T03:19:29.257 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.257 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:19:29.258 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:19:29.258 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.258 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.258 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:19:29.258 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:19:29.258 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:19:28.006820+0000" 2024-09-22T03:19:29.258 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.258 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:19:29.258 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:19:29.258 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.258 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.258 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:19:29.258 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:19:29.259 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.259 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.259 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:19:29.259 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:19:29.259 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.259 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.259 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:19:29.259 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:19:29.259 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:19:28.006904+0000" 2024-09-22T03:19:29.259 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.259 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:19:29.259 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:19:29.259 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.260 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.260 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:19:29.260 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:19:29.260 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:19:28.007953+0000" 2024-09-22T03:19:29.260 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.260 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:19:29.260 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:19:29.260 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.260 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.260 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:19:29.260 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:19:29.260 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.260 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.260 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:19:29.261 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:19:29.261 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.261 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.261 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:19:29.261 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:19:29.261 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:19:29.120939+0000" 2024-09-22T03:19:29.261 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.261 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:19:29.261 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:19:29.261 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:19:28.008373+0000" 2024-09-22T03:19:29.261 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.261 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:19:29.261 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:19:29.262 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.262 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.262 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:19:29.262 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:19:29.262 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.262 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.262 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:19:29.262 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:19:29.262 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.262 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.262 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:19:29.262 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:19:29.262 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:19:28.008613+0000" 2024-09-22T03:19:29.263 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.263 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:19:29.263 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:19:29.263 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.263 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.263 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:19:29.263 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:19:29.263 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:19:28.008720+0000" 2024-09-22T03:19:29.263 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.263 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:19:29.263 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:19:29.263 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.263 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.263 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:19:29.264 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:19:29.264 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:19:29.059621+0000" 2024-09-22T03:19:29.264 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.264 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:19:29.264 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:19:29.264 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.264 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.264 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:19:29.264 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:19:29.264 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.264 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.264 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:19:29.264 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:19:29.265 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.265 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.265 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:19:29.265 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:19:29.265 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.265 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.265 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:19:29.265 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:19:29.265 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.265 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.265 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:19:29.265 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:19:29.265 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.266 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.266 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:19:29.266 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:19:29.266 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.266 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.266 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:19:29.266 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:19:29.266 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:19:29.059937+0000" 2024-09-22T03:19:29.266 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.266 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:19:29.266 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:19:29.266 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.266 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.267 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:19:29.267 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:19:29.267 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.267 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.267 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:19:29.267 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:19:29.267 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.267 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.267 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:19:29.267 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:19:29.267 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:19:29.070884+0000" 2024-09-22T03:19:29.267 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.267 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:19:29.267 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:19:29.268 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.268 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.268 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:19:29.268 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:19:29.268 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.268 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.268 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:19:29.268 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:19:29.268 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.268 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.268 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:19:29.268 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:19:29.268 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:19:29.119848+0000" 2024-09-22T03:19:29.268 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.268 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:19:29.269 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:19:29.269 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.269 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:19:29.269 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:19:29.269 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:19:29.269 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:19:29.269 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:19:29.269 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:19:29.269 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:19:29.269 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-22T03:19:29.269 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-22T03:19:29.271 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:19:29.183+0000 7f9c1d757640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:19:29.271 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:19:29.189+0000 7f9c1d757640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:19:30.069 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 22 16 362 346 0.122856 0.122869 0.0513893 1.01687 2024-09-22T03:19:31.069 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 23 16 703 687 0.23333 2.66406 0.0638445 0.53507 2024-09-22T03:19:31.382 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-22T03:19:31.519 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:19:31.515+0000 7f2db122c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:19:31.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:19:31.522+0000 7f2db122c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:19:32.069 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 24 16 1052 1036 0.337203 2.72656 0.0704464 0.370151 2024-09-22T03:19:33.069 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 25 16 1400 1384 0.432453 2.71875 0.0388708 0.288817 2024-09-22T03:19:33.695 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-22T03:19:33.831 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:19:33.828+0000 7f5592997640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:19:33.835 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:19:33.834+0000 7f5592997640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:19:34.069 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 26 16 1760 1744 0.523982 2.8125 0.047241 0.238257 2024-09-22T03:19:34.272 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 2, 3, 1] 2024-09-22T03:19:34.272 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-22T03:19:34.272 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-09-22T03:19:34.272 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-09-22T03:19:34.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:19:34.404+0000 7fb134cd7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:19:34.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:19:34.413+0000 7fb134cd7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:19:34.470 INFO:teuthology.orchestra.run.smithi153.stderr:no valid command found; 10 closest matches: 2024-09-22T03:19:34.471 INFO:teuthology.orchestra.run.smithi153.stderr:config get 2024-09-22T03:19:34.471 INFO:teuthology.orchestra.run.smithi153.stderr:config help 2024-09-22T03:19:34.471 INFO:teuthology.orchestra.run.smithi153.stderr:config set ... 2024-09-22T03:19:34.471 INFO:teuthology.orchestra.run.smithi153.stderr:config show 2024-09-22T03:19:34.471 INFO:teuthology.orchestra.run.smithi153.stderr:deep_scrub 2024-09-22T03:19:34.471 INFO:teuthology.orchestra.run.smithi153.stderr:dump_historic_ops 2024-09-22T03:19:34.471 INFO:teuthology.orchestra.run.smithi153.stderr:dump_historic_slow_ops 2024-09-22T03:19:34.471 INFO:teuthology.orchestra.run.smithi153.stderr:dump_metrics [] 2024-09-22T03:19:34.471 INFO:teuthology.orchestra.run.smithi153.stderr:dump_ops_in_flight 2024-09-22T03:19:34.471 INFO:teuthology.orchestra.run.smithi153.stderr:dump_pgstate_history 2024-09-22T03:19:34.471 INFO:teuthology.orchestra.run.smithi153.stderr:Error EINVAL: invalid command 2024-09-22T03:19:34.474 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-22T03:19:34.474 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-22T03:19:34.474 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-09-22T03:19:34.618 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:19:34.610+0000 7f151e357640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:19:34.623 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:19:34.621+0000 7f151e357640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:19:34.647 INFO:teuthology.orchestra.run.smithi153.stderr:no valid command found; 10 closest matches: 2024-09-22T03:19:34.647 INFO:teuthology.orchestra.run.smithi153.stderr:config get 2024-09-22T03:19:34.647 INFO:teuthology.orchestra.run.smithi153.stderr:config help 2024-09-22T03:19:34.647 INFO:teuthology.orchestra.run.smithi153.stderr:config set ... 2024-09-22T03:19:34.647 INFO:teuthology.orchestra.run.smithi153.stderr:config show 2024-09-22T03:19:34.648 INFO:teuthology.orchestra.run.smithi153.stderr:deep_scrub 2024-09-22T03:19:34.648 INFO:teuthology.orchestra.run.smithi153.stderr:dump_historic_ops 2024-09-22T03:19:34.648 INFO:teuthology.orchestra.run.smithi153.stderr:dump_historic_slow_ops 2024-09-22T03:19:34.648 INFO:teuthology.orchestra.run.smithi153.stderr:dump_metrics [] 2024-09-22T03:19:34.648 INFO:teuthology.orchestra.run.smithi153.stderr:dump_ops_in_flight 2024-09-22T03:19:34.648 INFO:teuthology.orchestra.run.smithi153.stderr:dump_pgstate_history 2024-09-22T03:19:34.648 INFO:teuthology.orchestra.run.smithi153.stderr:Error EINVAL: invalid command 2024-09-22T03:19:34.652 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-22T03:19:34.653 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-22T03:19:34.653 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-09-22T03:19:34.793 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:19:34.786+0000 7fc0d5357640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:19:34.797 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:19:34.795+0000 7fc0d5357640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:19:34.813 INFO:teuthology.orchestra.run.smithi153.stderr:no valid command found; 10 closest matches: 2024-09-22T03:19:34.813 INFO:teuthology.orchestra.run.smithi153.stderr:config get 2024-09-22T03:19:34.813 INFO:teuthology.orchestra.run.smithi153.stderr:config help 2024-09-22T03:19:34.813 INFO:teuthology.orchestra.run.smithi153.stderr:config set ... 2024-09-22T03:19:34.813 INFO:teuthology.orchestra.run.smithi153.stderr:config show 2024-09-22T03:19:34.813 INFO:teuthology.orchestra.run.smithi153.stderr:deep_scrub 2024-09-22T03:19:34.813 INFO:teuthology.orchestra.run.smithi153.stderr:dump_historic_ops 2024-09-22T03:19:34.814 INFO:teuthology.orchestra.run.smithi153.stderr:dump_historic_slow_ops 2024-09-22T03:19:34.814 INFO:teuthology.orchestra.run.smithi153.stderr:dump_metrics [] 2024-09-22T03:19:34.814 INFO:teuthology.orchestra.run.smithi153.stderr:dump_ops_in_flight 2024-09-22T03:19:34.814 INFO:teuthology.orchestra.run.smithi153.stderr:dump_pgstate_history 2024-09-22T03:19:34.814 INFO:teuthology.orchestra.run.smithi153.stderr:Error EINVAL: invalid command 2024-09-22T03:19:34.816 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-22T03:19:34.817 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-22T03:19:34.817 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-09-22T03:19:34.949 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:19:34.942+0000 7f7858157640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:19:34.953 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:19:34.951+0000 7f7858157640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:19:35.012 INFO:teuthology.orchestra.run.smithi153.stderr:no valid command found; 10 closest matches: 2024-09-22T03:19:35.012 INFO:teuthology.orchestra.run.smithi153.stderr:config get 2024-09-22T03:19:35.012 INFO:teuthology.orchestra.run.smithi153.stderr:config help 2024-09-22T03:19:35.012 INFO:teuthology.orchestra.run.smithi153.stderr:config set ... 2024-09-22T03:19:35.012 INFO:teuthology.orchestra.run.smithi153.stderr:config show 2024-09-22T03:19:35.012 INFO:teuthology.orchestra.run.smithi153.stderr:deep_scrub 2024-09-22T03:19:35.012 INFO:teuthology.orchestra.run.smithi153.stderr:dump_historic_ops 2024-09-22T03:19:35.012 INFO:teuthology.orchestra.run.smithi153.stderr:dump_historic_slow_ops 2024-09-22T03:19:35.012 INFO:teuthology.orchestra.run.smithi153.stderr:dump_metrics [] 2024-09-22T03:19:35.012 INFO:teuthology.orchestra.run.smithi153.stderr:dump_ops_in_flight 2024-09-22T03:19:35.012 INFO:teuthology.orchestra.run.smithi153.stderr:dump_pgstate_history 2024-09-22T03:19:35.012 INFO:teuthology.orchestra.run.smithi153.stderr:Error EINVAL: invalid command 2024-09-22T03:19:35.015 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-22T03:19:35.016 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-22T03:19:35.069 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 27 16 2098 2082 0.602367 2.64062 0.0202917 0.207429 2024-09-22T03:19:36.003 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-22T03:19:36.069 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 28 16 2444 2428 0.677385 2.70312 0.0809014 0.184383 2024-09-22T03:19:36.136 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:19:36.132+0000 7fdad2e18640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:19:36.140 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:19:36.138+0000 7fdad2e18640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:19:37.069 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 29 16 2815 2799 0.753963 2.89844 0.0410462 0.165642 2024-09-22T03:19:38.070 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 30 16 3136 3120 0.812417 2.50781 0.0810382 0.153644 2024-09-22T03:19:38.298 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-22T03:19:38.443 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:19:38.440+0000 7fd587317640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:19:38.447 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:19:38.445+0000 7fd587317640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:19:39.070 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 31 16 3481 3465 0.873147 2.69531 0.0278021 0.143065 2024-09-22T03:19:40.017 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 2, 3, 1] 2024-09-22T03:19:40.017 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-22T03:19:40.017 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-09-22T03:19:40.017 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-09-22T03:19:40.070 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 32 16 3839 3823 0.933255 2.79688 0.0480979 0.133843 2024-09-22T03:19:40.160 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:19:40.153+0000 7f0fd206c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:19:40.165 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:19:40.163+0000 7f0fd206c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:19:40.202 INFO:teuthology.orchestra.run.smithi153.stderr:no valid command found; 10 closest matches: 2024-09-22T03:19:40.202 INFO:teuthology.orchestra.run.smithi153.stderr:config get 2024-09-22T03:19:40.203 INFO:teuthology.orchestra.run.smithi153.stderr:config help 2024-09-22T03:19:40.203 INFO:teuthology.orchestra.run.smithi153.stderr:config set ... 2024-09-22T03:19:40.203 INFO:teuthology.orchestra.run.smithi153.stderr:config show 2024-09-22T03:19:40.203 INFO:teuthology.orchestra.run.smithi153.stderr:deep_scrub 2024-09-22T03:19:40.203 INFO:teuthology.orchestra.run.smithi153.stderr:dump_historic_ops 2024-09-22T03:19:40.203 INFO:teuthology.orchestra.run.smithi153.stderr:dump_historic_slow_ops 2024-09-22T03:19:40.203 INFO:teuthology.orchestra.run.smithi153.stderr:dump_metrics [] 2024-09-22T03:19:40.203 INFO:teuthology.orchestra.run.smithi153.stderr:dump_ops_in_flight 2024-09-22T03:19:40.203 INFO:teuthology.orchestra.run.smithi153.stderr:dump_pgstate_history 2024-09-22T03:19:40.203 INFO:teuthology.orchestra.run.smithi153.stderr:Error EINVAL: invalid command 2024-09-22T03:19:40.207 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-22T03:19:40.208 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-22T03:19:40.208 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-09-22T03:19:40.329 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:19:40.322+0000 7f3b13917640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:19:40.334 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:19:40.333+0000 7f3b13917640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:19:40.404 INFO:teuthology.orchestra.run.smithi153.stderr:no valid command found; 10 closest matches: 2024-09-22T03:19:40.405 INFO:teuthology.orchestra.run.smithi153.stderr:config get 2024-09-22T03:19:40.405 INFO:teuthology.orchestra.run.smithi153.stderr:config help 2024-09-22T03:19:40.405 INFO:teuthology.orchestra.run.smithi153.stderr:config set ... 2024-09-22T03:19:40.405 INFO:teuthology.orchestra.run.smithi153.stderr:config show 2024-09-22T03:19:40.405 INFO:teuthology.orchestra.run.smithi153.stderr:deep_scrub 2024-09-22T03:19:40.405 INFO:teuthology.orchestra.run.smithi153.stderr:dump_historic_ops 2024-09-22T03:19:40.405 INFO:teuthology.orchestra.run.smithi153.stderr:dump_historic_slow_ops 2024-09-22T03:19:40.405 INFO:teuthology.orchestra.run.smithi153.stderr:dump_metrics [] 2024-09-22T03:19:40.405 INFO:teuthology.orchestra.run.smithi153.stderr:dump_ops_in_flight 2024-09-22T03:19:40.405 INFO:teuthology.orchestra.run.smithi153.stderr:dump_pgstate_history 2024-09-22T03:19:40.405 INFO:teuthology.orchestra.run.smithi153.stderr:Error EINVAL: invalid command 2024-09-22T03:19:40.408 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-22T03:19:40.408 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-22T03:19:40.409 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-09-22T03:19:40.574 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:19:40.562+0000 7f7fe4b0e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:19:40.581 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:19:40.580+0000 7f7fe4b0e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:19:40.594 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-22T03:19:40.602 INFO:teuthology.orchestra.run.smithi153.stderr:no valid command found; 10 closest matches: 2024-09-22T03:19:40.602 INFO:teuthology.orchestra.run.smithi153.stderr:config get 2024-09-22T03:19:40.602 INFO:teuthology.orchestra.run.smithi153.stderr:config help 2024-09-22T03:19:40.602 INFO:teuthology.orchestra.run.smithi153.stderr:config set ... 2024-09-22T03:19:40.602 INFO:teuthology.orchestra.run.smithi153.stderr:config show 2024-09-22T03:19:40.602 INFO:teuthology.orchestra.run.smithi153.stderr:deep_scrub 2024-09-22T03:19:40.602 INFO:teuthology.orchestra.run.smithi153.stderr:dump_historic_ops 2024-09-22T03:19:40.603 INFO:teuthology.orchestra.run.smithi153.stderr:dump_historic_slow_ops 2024-09-22T03:19:40.603 INFO:teuthology.orchestra.run.smithi153.stderr:dump_metrics [] 2024-09-22T03:19:40.603 INFO:teuthology.orchestra.run.smithi153.stderr:dump_ops_in_flight 2024-09-22T03:19:40.603 INFO:teuthology.orchestra.run.smithi153.stderr:dump_pgstate_history 2024-09-22T03:19:40.603 INFO:teuthology.orchestra.run.smithi153.stderr:Error EINVAL: invalid command 2024-09-22T03:19:40.606 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-22T03:19:40.606 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-22T03:19:40.606 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-09-22T03:19:40.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:19:40.745+0000 7f2e17997640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:19:40.752 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:19:40.751+0000 7f2e17997640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:19:40.783 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:19:40.776+0000 7fdbd0858640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:19:40.787 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:19:40.786+0000 7fdbd0858640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:19:40.806 INFO:teuthology.orchestra.run.smithi153.stderr:no valid command found; 10 closest matches: 2024-09-22T03:19:40.806 INFO:teuthology.orchestra.run.smithi153.stderr:config get 2024-09-22T03:19:40.806 INFO:teuthology.orchestra.run.smithi153.stderr:config help 2024-09-22T03:19:40.806 INFO:teuthology.orchestra.run.smithi153.stderr:config set ... 2024-09-22T03:19:40.806 INFO:teuthology.orchestra.run.smithi153.stderr:config show 2024-09-22T03:19:40.807 INFO:teuthology.orchestra.run.smithi153.stderr:deep_scrub 2024-09-22T03:19:40.807 INFO:teuthology.orchestra.run.smithi153.stderr:dump_historic_ops 2024-09-22T03:19:40.807 INFO:teuthology.orchestra.run.smithi153.stderr:dump_historic_slow_ops 2024-09-22T03:19:40.807 INFO:teuthology.orchestra.run.smithi153.stderr:dump_metrics [] 2024-09-22T03:19:40.807 INFO:teuthology.orchestra.run.smithi153.stderr:dump_ops_in_flight 2024-09-22T03:19:40.807 INFO:teuthology.orchestra.run.smithi153.stderr:dump_pgstate_history 2024-09-22T03:19:40.807 INFO:teuthology.orchestra.run.smithi153.stderr:Error EINVAL: invalid command 2024-09-22T03:19:40.809 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-22T03:19:40.810 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-22T03:19:41.070 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 33 16 4154 4138 0.979541 2.46094 0.0783585 0.12744 2024-09-22T03:19:42.070 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 34 16 4513 4497 1.03321 2.80469 0.0533781 0.12087 2024-09-22T03:19:43.006 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-22T03:19:43.070 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 35 16 4857 4841 1.08047 2.6875 0.0330422 0.11558 2024-09-22T03:19:43.146 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:19:43.143+0000 7fcbb74ce640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:19:43.150 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:19:43.148+0000 7fcbb74ce640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:19:44.070 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 36 16 5186 5170 1.12185 2.57031 0.0348908 0.111362 2024-09-22T03:19:45.070 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 37 16 5516 5500 1.1612 2.57812 0.0448857 0.107596 2024-09-22T03:19:45.403 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-22T03:19:45.549 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:19:45.545+0000 7f9acd46c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:19:45.553 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:19:45.550+0000 7f9acd46c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:19:45.811 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 2, 3, 1] 2024-09-22T03:19:45.811 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-22T03:19:45.811 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-09-22T03:19:45.811 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-09-22T03:19:45.952 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:19:45.942+0000 7f7084397640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:19:45.956 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:19:45.953+0000 7f7084397640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:19:46.064 INFO:teuthology.orchestra.run.smithi153.stderr:no valid command found; 10 closest matches: 2024-09-22T03:19:46.064 INFO:teuthology.orchestra.run.smithi153.stderr:config get 2024-09-22T03:19:46.064 INFO:teuthology.orchestra.run.smithi153.stderr:config help 2024-09-22T03:19:46.065 INFO:teuthology.orchestra.run.smithi153.stderr:config set ... 2024-09-22T03:19:46.065 INFO:teuthology.orchestra.run.smithi153.stderr:config show 2024-09-22T03:19:46.065 INFO:teuthology.orchestra.run.smithi153.stderr:deep_scrub 2024-09-22T03:19:46.065 INFO:teuthology.orchestra.run.smithi153.stderr:dump_historic_ops 2024-09-22T03:19:46.065 INFO:teuthology.orchestra.run.smithi153.stderr:dump_historic_slow_ops 2024-09-22T03:19:46.065 INFO:teuthology.orchestra.run.smithi153.stderr:dump_metrics [] 2024-09-22T03:19:46.065 INFO:teuthology.orchestra.run.smithi153.stderr:dump_ops_in_flight 2024-09-22T03:19:46.065 INFO:teuthology.orchestra.run.smithi153.stderr:dump_pgstate_history 2024-09-22T03:19:46.065 INFO:teuthology.orchestra.run.smithi153.stderr:Error EINVAL: invalid command 2024-09-22T03:19:46.067 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-22T03:19:46.068 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-22T03:19:46.068 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-09-22T03:19:46.070 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 38 16 5846 5830 1.19848 2.57812 0.0666 0.104231 2024-09-22T03:19:46.198 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:19:46.190+0000 7ff92b84f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:19:46.202 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:19:46.200+0000 7ff92b84f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:19:46.234 INFO:teuthology.orchestra.run.smithi153.stderr:no valid command found; 10 closest matches: 2024-09-22T03:19:46.234 INFO:teuthology.orchestra.run.smithi153.stderr:config get 2024-09-22T03:19:46.234 INFO:teuthology.orchestra.run.smithi153.stderr:config help 2024-09-22T03:19:46.234 INFO:teuthology.orchestra.run.smithi153.stderr:config set ... 2024-09-22T03:19:46.234 INFO:teuthology.orchestra.run.smithi153.stderr:config show 2024-09-22T03:19:46.234 INFO:teuthology.orchestra.run.smithi153.stderr:deep_scrub 2024-09-22T03:19:46.235 INFO:teuthology.orchestra.run.smithi153.stderr:dump_historic_ops 2024-09-22T03:19:46.235 INFO:teuthology.orchestra.run.smithi153.stderr:dump_historic_slow_ops 2024-09-22T03:19:46.235 INFO:teuthology.orchestra.run.smithi153.stderr:dump_metrics [] 2024-09-22T03:19:46.235 INFO:teuthology.orchestra.run.smithi153.stderr:dump_ops_in_flight 2024-09-22T03:19:46.235 INFO:teuthology.orchestra.run.smithi153.stderr:dump_pgstate_history 2024-09-22T03:19:46.235 INFO:teuthology.orchestra.run.smithi153.stderr:Error EINVAL: invalid command 2024-09-22T03:19:46.237 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-22T03:19:46.237 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-22T03:19:46.237 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-09-22T03:19:46.360 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:19:46.351+0000 7f41d1ad7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:19:46.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:19:46.363+0000 7f41d1ad7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:19:46.395 INFO:teuthology.orchestra.run.smithi153.stderr:no valid command found; 10 closest matches: 2024-09-22T03:19:46.395 INFO:teuthology.orchestra.run.smithi153.stderr:config get 2024-09-22T03:19:46.395 INFO:teuthology.orchestra.run.smithi153.stderr:config help 2024-09-22T03:19:46.395 INFO:teuthology.orchestra.run.smithi153.stderr:config set ... 2024-09-22T03:19:46.395 INFO:teuthology.orchestra.run.smithi153.stderr:config show 2024-09-22T03:19:46.395 INFO:teuthology.orchestra.run.smithi153.stderr:deep_scrub 2024-09-22T03:19:46.396 INFO:teuthology.orchestra.run.smithi153.stderr:dump_historic_ops 2024-09-22T03:19:46.396 INFO:teuthology.orchestra.run.smithi153.stderr:dump_historic_slow_ops 2024-09-22T03:19:46.396 INFO:teuthology.orchestra.run.smithi153.stderr:dump_metrics [] 2024-09-22T03:19:46.396 INFO:teuthology.orchestra.run.smithi153.stderr:dump_ops_in_flight 2024-09-22T03:19:46.396 INFO:teuthology.orchestra.run.smithi153.stderr:dump_pgstate_history 2024-09-22T03:19:46.396 INFO:teuthology.orchestra.run.smithi153.stderr:Error EINVAL: invalid command 2024-09-22T03:19:46.399 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-22T03:19:46.400 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-22T03:19:46.400 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-09-22T03:19:46.546 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:19:46.534+0000 7fc93de2c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:19:46.551 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:19:46.548+0000 7fc93de2c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:19:46.584 INFO:teuthology.orchestra.run.smithi153.stderr:no valid command found; 10 closest matches: 2024-09-22T03:19:46.584 INFO:teuthology.orchestra.run.smithi153.stderr:config get 2024-09-22T03:19:46.585 INFO:teuthology.orchestra.run.smithi153.stderr:config help 2024-09-22T03:19:46.585 INFO:teuthology.orchestra.run.smithi153.stderr:config set ... 2024-09-22T03:19:46.585 INFO:teuthology.orchestra.run.smithi153.stderr:config show 2024-09-22T03:19:46.585 INFO:teuthology.orchestra.run.smithi153.stderr:deep_scrub 2024-09-22T03:19:46.585 INFO:teuthology.orchestra.run.smithi153.stderr:dump_historic_ops 2024-09-22T03:19:46.585 INFO:teuthology.orchestra.run.smithi153.stderr:dump_historic_slow_ops 2024-09-22T03:19:46.585 INFO:teuthology.orchestra.run.smithi153.stderr:dump_metrics [] 2024-09-22T03:19:46.585 INFO:teuthology.orchestra.run.smithi153.stderr:dump_ops_in_flight 2024-09-22T03:19:46.585 INFO:teuthology.orchestra.run.smithi153.stderr:dump_pgstate_history 2024-09-22T03:19:46.585 INFO:teuthology.orchestra.run.smithi153.stderr:Error EINVAL: invalid command 2024-09-22T03:19:46.588 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-22T03:19:46.588 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-22T03:19:47.070 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 39 16 6178 6162 1.23425 2.59375 0.0562204 0.101159 2024-09-22T03:19:47.666 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-22T03:19:47.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:19:47.802+0000 7f4cb8a4f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:19:47.810 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:19:47.808+0000 7f4cb8a4f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:19:48.070 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:2024-09-22T03:19:48.069900+0000 min lat: 0.00858012 max lat: 21.1302 avg lat: 0.0990565 2024-09-22T03:19:48.071 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-22T03:19:48.071 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 40 16 6473 6457 1.26101 2.30469 0.109776 0.0990565 2024-09-22T03:19:49.070 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 41 16 6808 6792 1.29408 2.61719 0.04472 0.0965183 2024-09-22T03:19:49.991 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-22T03:19:50.071 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 42 16 7144 7128 1.32576 2.625 0.0270389 0.0942369 2024-09-22T03:19:50.112 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:19:50.108+0000 7f95e2c2c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:19:50.116 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:19:50.113+0000 7f95e2c2c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:19:51.071 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 43 16 7448 7432 1.35016 2.375 0.0470943 0.0925152 2024-09-22T03:19:51.589 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 2, 3, 1] 2024-09-22T03:19:51.589 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-22T03:19:51.589 INFO:tasks.thrashosds.thrasher:Killing osd 0, live_osds are [0, 2, 3, 1] 2024-09-22T03:19:51.590 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-09-22T03:19:51.590 INFO:teuthology.orchestra.run:waiting for 300 2024-09-22T03:19:52.071 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 44 16 7622 7606 1.35037 1.35938 0.0232493 0.0915191 2024-09-22T03:19:52.294 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-22T03:19:52.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:19:52.396+0000 7fa738117640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:19:52.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:19:52.401+0000 7fa738117640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:19:53.071 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 45 16 7622 7606 1.32036 0 - 0.0915191 2024-09-22T03:19:54.071 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 46 16 7622 7606 1.29165 0 - 0.0915191 2024-09-22T03:19:55.071 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 47 16 7622 7606 1.26417 0 - 0.0915191 2024-09-22T03:19:56.071 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 48 16 7622 7606 1.23783 0 - 0.0915191 2024-09-22T03:19:56.356 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~0s 2024-09-22T03:19:57.071 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 49 16 7622 7606 1.21257 0 - 0.0915191 2024-09-22T03:19:57.691 INFO:tasks.ceph.osd.0:Stopped 2024-09-22T03:19:58.072 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 50 16 7622 7606 1.18832 0 - 0.0915191 2024-09-22T03:19:59.072 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 51 16 7622 7606 1.16502 0 - 0.0915191 2024-09-22T03:20:00.072 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 52 16 7622 7606 1.14261 0 - 0.0915191 2024-09-22T03:20:01.072 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 53 16 7622 7606 1.12105 0 - 0.0915191 2024-09-22T03:20:01.861 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 has been restored 2024-09-22T03:20:02.072 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 54 16 7622 7606 1.10029 0 - 0.0915191 2024-09-22T03:20:02.692 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [0] live_osds: [2, 3, 1] 2024-09-22T03:20:02.692 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-22T03:20:02.692 INFO:tasks.thrashosds.thrasher:Reviving osd 0 2024-09-22T03:20:02.692 INFO:tasks.ceph.osd.0:Restarting daemon 2024-09-22T03:20:02.692 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2>> /var/log/ceph/ceph-osd.0.log 2024-09-22T03:20:02.695 INFO:tasks.ceph.osd.0:Started 2024-09-22T03:20:02.695 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-22T03:20:02.818 INFO:teuthology.orchestra.run.smithi153.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-09-22T03:20:02.821 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-22T03:20:02.822 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-0, ['dump_ops_in_flight'] 2024-09-22T03:20:03.072 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 55 16 7622 7606 1.08028 0 - 0.0915191 2024-09-22T03:20:04.073 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 56 16 7622 7606 1.06099 0 - 0.0915191 2024-09-22T03:20:05.073 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 57 16 7622 7606 1.04238 0 - 0.0915191 2024-09-22T03:20:06.072 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 58 16 7622 7606 1.02441 0 - 0.0915191 2024-09-22T03:20:07.073 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 59 16 7622 7606 1.00704 0 - 0.0915191 2024-09-22T03:20:07.823 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-22T03:20:07.920 INFO:teuthology.orchestra.run.smithi153.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-09-22T03:20:07.923 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-22T03:20:07.923 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-0, ['dump_ops_in_flight'] 2024-09-22T03:20:08.073 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:2024-09-22T03:20:08.072286+0000 min lat: 0.00858012 max lat: 21.1302 avg lat: 0.0915191 2024-09-22T03:20:08.073 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-22T03:20:08.073 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 60 16 7622 7606 0.990261 0 - 0.0915191 2024-09-22T03:20:09.073 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 61 16 7622 7606 0.974027 0 - 0.0915191 2024-09-22T03:20:10.073 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 62 16 7622 7606 0.958316 0 - 0.0915191 2024-09-22T03:20:11.073 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 63 16 7622 7606 0.943105 0 - 0.0915191 2024-09-22T03:20:12.073 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 64 16 7622 7606 0.928369 0 - 0.0915191 2024-09-22T03:20:12.924 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-22T03:20:12.958 INFO:teuthology.orchestra.run.smithi153.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-22T03:20:13.070 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-22T03:20:13.070 INFO:teuthology.orchestra.run.smithi153.stdout: "ops_in_flight": [] 2024-09-22T03:20:13.070 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-22T03:20:13.073 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 65 16 7622 7606 0.914087 0 - 0.0915191 2024-09-22T03:20:13.085 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-22T03:20:13.086 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-22T03:20:14.073 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 66 16 7622 7606 0.900237 0 - 0.0915191 2024-09-22T03:20:15.073 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 67 16 7622 7606 0.886801 0 - 0.0915191 2024-09-22T03:20:15.087 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-22T03:20:15.213 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:20:15.210+0000 7fd7b34a9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:20:15.218 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:20:15.216+0000 7fd7b34a9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:20:16.074 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 68 16 7921 7905 0.908108 0.0973307 0.0327732 0.137557 2024-09-22T03:20:17.074 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 69 16 8219 8203 0.928685 2.32812 0.0366529 0.134535 2024-09-22T03:20:17.420 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-22T03:20:17.543 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:20:17.540+0000 7f5f0cefa640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:20:17.547 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:20:17.546+0000 7f5f0cefa640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:20:18.074 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 70 16 8536 8520 0.950794 2.47656 0.0627009 0.13141 2024-09-22T03:20:18.081 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [2, 3, 1, 0] 2024-09-22T03:20:18.081 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-22T03:20:18.081 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-09-22T03:20:18.082 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-09-22T03:20:18.236 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:20:18.229+0000 7f8ea9197640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:20:18.240 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:20:18.238+0000 7f8ea9197640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:20:18.271 INFO:teuthology.orchestra.run.smithi153.stderr:no valid command found; 10 closest matches: 2024-09-22T03:20:18.271 INFO:teuthology.orchestra.run.smithi153.stderr:config get 2024-09-22T03:20:18.271 INFO:teuthology.orchestra.run.smithi153.stderr:config help 2024-09-22T03:20:18.271 INFO:teuthology.orchestra.run.smithi153.stderr:config set ... 2024-09-22T03:20:18.272 INFO:teuthology.orchestra.run.smithi153.stderr:config show 2024-09-22T03:20:18.272 INFO:teuthology.orchestra.run.smithi153.stderr:deep_scrub 2024-09-22T03:20:18.272 INFO:teuthology.orchestra.run.smithi153.stderr:dump_historic_ops 2024-09-22T03:20:18.272 INFO:teuthology.orchestra.run.smithi153.stderr:dump_historic_slow_ops 2024-09-22T03:20:18.272 INFO:teuthology.orchestra.run.smithi153.stderr:dump_metrics [] 2024-09-22T03:20:18.272 INFO:teuthology.orchestra.run.smithi153.stderr:dump_ops_in_flight 2024-09-22T03:20:18.272 INFO:teuthology.orchestra.run.smithi153.stderr:dump_pgstate_history 2024-09-22T03:20:18.272 INFO:teuthology.orchestra.run.smithi153.stderr:Error EINVAL: invalid command 2024-09-22T03:20:18.275 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-22T03:20:18.275 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-22T03:20:18.275 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-09-22T03:20:18.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:20:18.390+0000 7f26bfcd7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:20:18.400 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:20:18.399+0000 7f26bfcd7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:20:18.457 INFO:teuthology.orchestra.run.smithi153.stderr:no valid command found; 10 closest matches: 2024-09-22T03:20:18.457 INFO:teuthology.orchestra.run.smithi153.stderr:config get 2024-09-22T03:20:18.457 INFO:teuthology.orchestra.run.smithi153.stderr:config help 2024-09-22T03:20:18.457 INFO:teuthology.orchestra.run.smithi153.stderr:config set ... 2024-09-22T03:20:18.457 INFO:teuthology.orchestra.run.smithi153.stderr:config show 2024-09-22T03:20:18.457 INFO:teuthology.orchestra.run.smithi153.stderr:deep_scrub 2024-09-22T03:20:18.458 INFO:teuthology.orchestra.run.smithi153.stderr:dump_historic_ops 2024-09-22T03:20:18.458 INFO:teuthology.orchestra.run.smithi153.stderr:dump_historic_slow_ops 2024-09-22T03:20:18.458 INFO:teuthology.orchestra.run.smithi153.stderr:dump_metrics [] 2024-09-22T03:20:18.458 INFO:teuthology.orchestra.run.smithi153.stderr:dump_ops_in_flight 2024-09-22T03:20:18.458 INFO:teuthology.orchestra.run.smithi153.stderr:dump_pgstate_history 2024-09-22T03:20:18.458 INFO:teuthology.orchestra.run.smithi153.stderr:Error EINVAL: invalid command 2024-09-22T03:20:18.460 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-22T03:20:18.460 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-22T03:20:18.461 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-09-22T03:20:18.574 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:20:18.567+0000 7effc82b2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:20:18.578 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:20:18.576+0000 7effc82b2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:20:18.599 INFO:teuthology.orchestra.run.smithi153.stderr:no valid command found; 10 closest matches: 2024-09-22T03:20:18.599 INFO:teuthology.orchestra.run.smithi153.stderr:config get 2024-09-22T03:20:18.599 INFO:teuthology.orchestra.run.smithi153.stderr:config help 2024-09-22T03:20:18.599 INFO:teuthology.orchestra.run.smithi153.stderr:config set ... 2024-09-22T03:20:18.599 INFO:teuthology.orchestra.run.smithi153.stderr:config show 2024-09-22T03:20:18.599 INFO:teuthology.orchestra.run.smithi153.stderr:deep_scrub 2024-09-22T03:20:18.599 INFO:teuthology.orchestra.run.smithi153.stderr:dump_historic_ops 2024-09-22T03:20:18.599 INFO:teuthology.orchestra.run.smithi153.stderr:dump_historic_slow_ops 2024-09-22T03:20:18.600 INFO:teuthology.orchestra.run.smithi153.stderr:dump_metrics [] 2024-09-22T03:20:18.600 INFO:teuthology.orchestra.run.smithi153.stderr:dump_ops_in_flight 2024-09-22T03:20:18.600 INFO:teuthology.orchestra.run.smithi153.stderr:dump_pgstate_history 2024-09-22T03:20:18.600 INFO:teuthology.orchestra.run.smithi153.stderr:Error EINVAL: invalid command 2024-09-22T03:20:18.602 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-22T03:20:18.602 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-22T03:20:18.602 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-09-22T03:20:18.724 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:20:18.716+0000 7f291f997640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:20:18.729 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:20:18.728+0000 7f291f997640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:20:18.749 INFO:teuthology.orchestra.run.smithi153.stderr:no valid command found; 10 closest matches: 2024-09-22T03:20:18.749 INFO:teuthology.orchestra.run.smithi153.stderr:config get 2024-09-22T03:20:18.749 INFO:teuthology.orchestra.run.smithi153.stderr:config help 2024-09-22T03:20:18.749 INFO:teuthology.orchestra.run.smithi153.stderr:config set ... 2024-09-22T03:20:18.749 INFO:teuthology.orchestra.run.smithi153.stderr:config show 2024-09-22T03:20:18.749 INFO:teuthology.orchestra.run.smithi153.stderr:deep_scrub 2024-09-22T03:20:18.749 INFO:teuthology.orchestra.run.smithi153.stderr:dump_historic_ops 2024-09-22T03:20:18.749 INFO:teuthology.orchestra.run.smithi153.stderr:dump_historic_slow_ops 2024-09-22T03:20:18.749 INFO:teuthology.orchestra.run.smithi153.stderr:dump_metrics [] 2024-09-22T03:20:18.749 INFO:teuthology.orchestra.run.smithi153.stderr:dump_ops_in_flight 2024-09-22T03:20:18.749 INFO:teuthology.orchestra.run.smithi153.stderr:dump_pgstate_history 2024-09-22T03:20:18.749 INFO:teuthology.orchestra.run.smithi153.stderr:Error EINVAL: invalid command 2024-09-22T03:20:18.751 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-22T03:20:18.752 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-22T03:20:19.074 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 71 16 8818 8802 0.96843 2.20312 0.0445322 0.129003 2024-09-22T03:20:19.725 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-22T03:20:19.846 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:20:19.842+0000 7f1f3b02c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:20:19.849 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:20:19.847+0000 7f1f3b02c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:20:20.074 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 72 16 9138 9122 0.989699 2.5 0.036021 0.126256 2024-09-22T03:20:21.074 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 73 16 9457 9441 1.01028 2.49219 0.0239162 0.123675 2024-09-22T03:20:22.002 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-22T03:20:22.074 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 74 16 9804 9788 1.03326 2.71094 0.0257946 0.120908 2024-09-22T03:20:22.133 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:20:22.130+0000 7f9834d0e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:20:22.137 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:20:22.136+0000 7f9834d0e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:20:23.074 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 75 16 10109 10093 1.05125 2.38281 0.0724991 0.118841 2024-09-22T03:20:23.752 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [2, 3, 1, 0] 2024-09-22T03:20:23.752 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-22T03:20:23.752 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-09-22T03:20:23.753 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-09-22T03:20:23.872 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:20:23.865+0000 7fe723d17640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:20:23.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:20:23.874+0000 7fe723d17640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:20:23.934 INFO:teuthology.orchestra.run.smithi153.stderr:no valid command found; 10 closest matches: 2024-09-22T03:20:23.934 INFO:teuthology.orchestra.run.smithi153.stderr:config get 2024-09-22T03:20:23.934 INFO:teuthology.orchestra.run.smithi153.stderr:config help 2024-09-22T03:20:23.934 INFO:teuthology.orchestra.run.smithi153.stderr:config set ... 2024-09-22T03:20:23.934 INFO:teuthology.orchestra.run.smithi153.stderr:config show 2024-09-22T03:20:23.934 INFO:teuthology.orchestra.run.smithi153.stderr:deep_scrub 2024-09-22T03:20:23.934 INFO:teuthology.orchestra.run.smithi153.stderr:dump_historic_ops 2024-09-22T03:20:23.935 INFO:teuthology.orchestra.run.smithi153.stderr:dump_historic_slow_ops 2024-09-22T03:20:23.935 INFO:teuthology.orchestra.run.smithi153.stderr:dump_metrics [] 2024-09-22T03:20:23.935 INFO:teuthology.orchestra.run.smithi153.stderr:dump_ops_in_flight 2024-09-22T03:20:23.935 INFO:teuthology.orchestra.run.smithi153.stderr:dump_pgstate_history 2024-09-22T03:20:23.935 INFO:teuthology.orchestra.run.smithi153.stderr:Error EINVAL: invalid command 2024-09-22T03:20:23.937 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-22T03:20:23.937 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-22T03:20:23.937 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-09-22T03:20:24.074 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 76 16 10436 10420 1.07103 2.55469 0.0367123 0.116649 2024-09-22T03:20:24.096 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:20:24.088+0000 7f3d01a2c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:20:24.099 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:20:24.098+0000 7f3d01a2c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:20:24.114 INFO:teuthology.orchestra.run.smithi153.stderr:no valid command found; 10 closest matches: 2024-09-22T03:20:24.114 INFO:teuthology.orchestra.run.smithi153.stderr:config get 2024-09-22T03:20:24.114 INFO:teuthology.orchestra.run.smithi153.stderr:config help 2024-09-22T03:20:24.114 INFO:teuthology.orchestra.run.smithi153.stderr:config set ... 2024-09-22T03:20:24.114 INFO:teuthology.orchestra.run.smithi153.stderr:config show 2024-09-22T03:20:24.114 INFO:teuthology.orchestra.run.smithi153.stderr:deep_scrub 2024-09-22T03:20:24.114 INFO:teuthology.orchestra.run.smithi153.stderr:dump_historic_ops 2024-09-22T03:20:24.115 INFO:teuthology.orchestra.run.smithi153.stderr:dump_historic_slow_ops 2024-09-22T03:20:24.115 INFO:teuthology.orchestra.run.smithi153.stderr:dump_metrics [] 2024-09-22T03:20:24.115 INFO:teuthology.orchestra.run.smithi153.stderr:dump_ops_in_flight 2024-09-22T03:20:24.115 INFO:teuthology.orchestra.run.smithi153.stderr:dump_pgstate_history 2024-09-22T03:20:24.115 INFO:teuthology.orchestra.run.smithi153.stderr:Error EINVAL: invalid command 2024-09-22T03:20:24.118 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-22T03:20:24.118 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-22T03:20:24.118 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-09-22T03:20:24.239 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:20:24.232+0000 7f717e8d7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:20:24.243 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:20:24.242+0000 7f717e8d7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:20:24.257 INFO:teuthology.orchestra.run.smithi153.stderr:no valid command found; 10 closest matches: 2024-09-22T03:20:24.257 INFO:teuthology.orchestra.run.smithi153.stderr:config get 2024-09-22T03:20:24.257 INFO:teuthology.orchestra.run.smithi153.stderr:config help 2024-09-22T03:20:24.257 INFO:teuthology.orchestra.run.smithi153.stderr:config set ... 2024-09-22T03:20:24.257 INFO:teuthology.orchestra.run.smithi153.stderr:config show 2024-09-22T03:20:24.257 INFO:teuthology.orchestra.run.smithi153.stderr:deep_scrub 2024-09-22T03:20:24.257 INFO:teuthology.orchestra.run.smithi153.stderr:dump_historic_ops 2024-09-22T03:20:24.258 INFO:teuthology.orchestra.run.smithi153.stderr:dump_historic_slow_ops 2024-09-22T03:20:24.258 INFO:teuthology.orchestra.run.smithi153.stderr:dump_metrics [] 2024-09-22T03:20:24.258 INFO:teuthology.orchestra.run.smithi153.stderr:dump_ops_in_flight 2024-09-22T03:20:24.258 INFO:teuthology.orchestra.run.smithi153.stderr:dump_pgstate_history 2024-09-22T03:20:24.258 INFO:teuthology.orchestra.run.smithi153.stderr:Error EINVAL: invalid command 2024-09-22T03:20:24.260 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-22T03:20:24.260 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-22T03:20:24.261 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-09-22T03:20:24.289 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-22T03:20:24.418 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:20:24.410+0000 7f2db4997640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:20:24.422 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:20:24.420+0000 7f2db4997640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:20:24.433 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:20:24.429+0000 7fbc5d46c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:20:24.437 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:20:24.435+0000 7fbc57fff640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:20:24.450 INFO:teuthology.orchestra.run.smithi153.stderr:no valid command found; 10 closest matches: 2024-09-22T03:20:24.450 INFO:teuthology.orchestra.run.smithi153.stderr:config get 2024-09-22T03:20:24.450 INFO:teuthology.orchestra.run.smithi153.stderr:config help 2024-09-22T03:20:24.450 INFO:teuthology.orchestra.run.smithi153.stderr:config set ... 2024-09-22T03:20:24.450 INFO:teuthology.orchestra.run.smithi153.stderr:config show 2024-09-22T03:20:24.450 INFO:teuthology.orchestra.run.smithi153.stderr:deep_scrub 2024-09-22T03:20:24.451 INFO:teuthology.orchestra.run.smithi153.stderr:dump_historic_ops 2024-09-22T03:20:24.451 INFO:teuthology.orchestra.run.smithi153.stderr:dump_historic_slow_ops 2024-09-22T03:20:24.451 INFO:teuthology.orchestra.run.smithi153.stderr:dump_metrics [] 2024-09-22T03:20:24.451 INFO:teuthology.orchestra.run.smithi153.stderr:dump_ops_in_flight 2024-09-22T03:20:24.451 INFO:teuthology.orchestra.run.smithi153.stderr:dump_pgstate_history 2024-09-22T03:20:24.451 INFO:teuthology.orchestra.run.smithi153.stderr:Error EINVAL: invalid command 2024-09-22T03:20:24.454 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-22T03:20:24.455 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-22T03:20:25.074 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 77 16 10717 10701 1.08562 2.19531 0.0692709 0.115095 2024-09-22T03:20:26.074 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 78 16 11024 11008 1.10245 2.39844 0.0340633 0.113341 2024-09-22T03:20:26.603 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-22T03:20:26.734 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:20:26.730+0000 7f01f9d57640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:20:26.737 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:20:26.736+0000 7f01f9d57640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:20:27.074 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 79 16 11327 11311 1.11846 2.36719 0.0759058 0.111711 2024-09-22T03:20:28.075 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:2024-09-22T03:20:28.074060+0000 min lat: 0.00858012 max lat: 23.6203 avg lat: 0.109958 2024-09-22T03:20:28.075 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-22T03:20:28.075 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 80 16 11654 11638 1.13641 2.55469 0.0303209 0.109958 2024-09-22T03:20:28.895 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-22T03:20:29.025 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:20:29.022+0000 7f2988b97640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:20:29.030 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:20:29.028+0000 7f2988b97640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:20:29.074 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 81 16 11974 11958 1.15324 2.5 0.0466898 0.10834 2024-09-22T03:20:29.456 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [2, 3, 1, 0] 2024-09-22T03:20:29.456 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-22T03:20:29.456 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-22T03:20:29.591 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:20:29.582+0000 7f355f4d7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:20:29.595 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:20:29.594+0000 7f355f4d7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:20:29.874 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-22T03:20:29.874 INFO:teuthology.orchestra.run.smithi153.stderr:dumped all 2024-09-22T03:20:29.887 INFO:teuthology.orchestra.run.smithi153.stdout:{"pg_ready":true,"pg_map":{"version":510,"stamp":"2024-09-22T03:20:28.265284+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":93463091,"num_objects":1422,"num_object_clones":0,"num_object_copies":4266,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":11476,"num_write_kb":92592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":90112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":11421,"ondisk_log_size":11421,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":278660,"kb_used_data":139140,"kb_used_omap":0,"kb_used_meta":139456,"kb_avail":377208700,"statfs":{"total":386547056640,"available":386261708800,"internally_reserved":0,"allocated":142479360,"data_stored":278682765,"data_compressed":14283876,"data_compressed_allocated":137572352,"data_compressed_original":276676608,"omap_allocated":0,"internal_metadata":142802944},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":25255936,"num_objects":386,"num_object_clones":0,"num_object_copies":1158,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3083,"num_write_kb":24664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001738"},"pg_stats":[{"pgid":"9.7","version":"144'725","reported_seq":763,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-22T03:20:28.110863+0000","last_change":"2024-09-22T03:20:15.072786+0000","last_active":"2024-09-22T03:20:28.110863+0000","last_peered":"2024-09-22T03:20:28.110863+0000","last_clean":"2024-09-22T03:20:28.110863+0000","last_became_active":"2024-09-22T03:20:15.066379+0000","last_became_peered":"2024-09-22T03:20:15.066379+0000","last_unstale":"2024-09-22T03:20:28.110863+0000","last_undegraded":"2024-09-22T03:20:28.110863+0000","last_fullsized":"2024-09-22T03:20:28.110863+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:19:05.974214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:19:05.974214+0000","last_clean_scrub_stamp":"2024-09-22T03:19:05.974214+0000","objects_scrubbed":0,"log_size":725,"log_dups_size":0,"ondisk_log_size":725,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5947392,"num_objects":91,"num_object_clones":0,"num_object_copies":273,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":726,"num_write_kb":5808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.6","version":"144'703","reported_seq":737,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-22T03:20:25.535443+0000","last_change":"2024-09-22T03:19:29.016035+0000","last_active":"2024-09-22T03:20:25.535443+0000","last_peered":"2024-09-22T03:20:25.535443+0000","last_clean":"2024-09-22T03:20:25.535443+0000","last_became_active":"2024-09-22T03:19:29.005275+0000","last_became_peered":"2024-09-22T03:19:29.005275+0000","last_unstale":"2024-09-22T03:20:25.535443+0000","last_undegraded":"2024-09-22T03:20:25.535443+0000","last_fullsized":"2024-09-22T03:20:25.535443+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:19:05.974214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:19:05.974214+0000","last_clean_scrub_stamp":"2024-09-22T03:19:05.974214+0000","objects_scrubbed":0,"log_size":703,"log_dups_size":0,"ondisk_log_size":703,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5767168,"num_objects":88,"num_object_clones":0,"num_object_copies":264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":704,"num_write_kb":5632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.5","version":"144'647","reported_seq":685,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-22T03:20:25.995776+0000","last_change":"2024-09-22T03:20:15.046805+0000","last_active":"2024-09-22T03:20:25.995776+0000","last_peered":"2024-09-22T03:20:25.995776+0000","last_clean":"2024-09-22T03:20:25.995776+0000","last_became_active":"2024-09-22T03:20:15.042430+0000","last_became_peered":"2024-09-22T03:20:15.042430+0000","last_unstale":"2024-09-22T03:20:25.995776+0000","last_undegraded":"2024-09-22T03:20:25.995776+0000","last_fullsized":"2024-09-22T03:20:25.995776+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:19:05.974214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:19:05.974214+0000","last_clean_scrub_stamp":"2024-09-22T03:19:05.974214+0000","objects_scrubbed":0,"log_size":647,"log_dups_size":0,"ondisk_log_size":647,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5308416,"num_objects":81,"num_object_clones":0,"num_object_copies":243,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":648,"num_write_kb":5184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.4","version":"144'615","reported_seq":653,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-22T03:20:26.474533+0000","last_change":"2024-09-22T03:20:15.029557+0000","last_active":"2024-09-22T03:20:26.474533+0000","last_peered":"2024-09-22T03:20:26.474533+0000","last_clean":"2024-09-22T03:20:26.474533+0000","last_became_active":"2024-09-22T03:20:15.025087+0000","last_became_peered":"2024-09-22T03:20:15.025087+0000","last_unstale":"2024-09-22T03:20:26.474533+0000","last_undegraded":"2024-09-22T03:20:26.474533+0000","last_fullsized":"2024-09-22T03:20:26.474533+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:19:05.974214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:19:05.974214+0000","last_clean_scrub_stamp":"2024-09-22T03:19:05.974214+0000","objects_scrubbed":0,"log_size":615,"log_dups_size":0,"ondisk_log_size":615,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5046272,"num_objects":77,"num_object_clones":0,"num_object_copies":231,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":616,"num_write_kb":4928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3","version":"144'799","reported_seq":837,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-22T03:20:25.032927+0000","last_change":"2024-09-22T03:20:15.043854+0000","last_active":"2024-09-22T03:20:25.032927+0000","last_peered":"2024-09-22T03:20:25.032927+0000","last_clean":"2024-09-22T03:20:25.032927+0000","last_became_active":"2024-09-22T03:20:15.031653+0000","last_became_peered":"2024-09-22T03:20:15.031653+0000","last_unstale":"2024-09-22T03:20:25.032927+0000","last_undegraded":"2024-09-22T03:20:25.032927+0000","last_fullsized":"2024-09-22T03:20:25.032927+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:19:05.974214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:19:05.974214+0000","last_clean_scrub_stamp":"2024-09-22T03:19:05.974214+0000","objects_scrubbed":0,"log_size":799,"log_dups_size":0,"ondisk_log_size":799,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6553600,"num_objects":100,"num_object_clones":0,"num_object_copies":300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":800,"num_write_kb":6400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2","version":"144'639","reported_seq":674,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-22T03:20:27.269997+0000","last_change":"2024-09-22T03:20:15.028458+0000","last_active":"2024-09-22T03:20:27.269997+0000","last_peered":"2024-09-22T03:20:27.269997+0000","last_clean":"2024-09-22T03:20:27.269997+0000","last_became_active":"2024-09-22T03:20:15.024384+0000","last_became_peered":"2024-09-22T03:20:15.024384+0000","last_unstale":"2024-09-22T03:20:27.269997+0000","last_undegraded":"2024-09-22T03:20:27.269997+0000","last_fullsized":"2024-09-22T03:20:27.269997+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:19:05.974214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:19:05.974214+0000","last_clean_scrub_stamp":"2024-09-22T03:19:05.974214+0000","objects_scrubbed":0,"log_size":639,"log_dups_size":0,"ondisk_log_size":639,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5242880,"num_objects":80,"num_object_clones":0,"num_object_copies":240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":640,"num_write_kb":5120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"144'855","reported_seq":897,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-22T03:20:28.086319+0000","last_change":"2024-09-22T03:20:15.071552+0000","last_active":"2024-09-22T03:20:28.086319+0000","last_peered":"2024-09-22T03:20:28.086319+0000","last_clean":"2024-09-22T03:20:28.086319+0000","last_became_active":"2024-09-22T03:20:15.063905+0000","last_became_peered":"2024-09-22T03:20:15.063905+0000","last_unstale":"2024-09-22T03:20:28.086319+0000","last_undegraded":"2024-09-22T03:20:28.086319+0000","last_fullsized":"2024-09-22T03:20:28.086319+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:19:05.974214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:19:05.974214+0000","last_clean_scrub_stamp":"2024-09-22T03:19:05.974214+0000","objects_scrubbed":0,"log_size":855,"log_dups_size":0,"ondisk_log_size":855,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7012352,"num_objects":107,"num_object_clones":0,"num_object_copies":321,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":856,"num_write_kb":6848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":8192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.0","version":"144'631","reported_seq":670,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-22T03:20:26.068927+0000","last_change":"2024-09-22T03:20:15.012888+0000","last_active":"2024-09-22T03:20:26.068927+0000","last_peered":"2024-09-22T03:20:26.068927+0000","last_clean":"2024-09-22T03:20:26.068927+0000","last_became_active":"2024-09-22T03:20:15.005060+0000","last_became_peered":"2024-09-22T03:20:15.005060+0000","last_unstale":"2024-09-22T03:20:26.068927+0000","last_undegraded":"2024-09-22T03:20:26.068927+0000","last_fullsized":"2024-09-22T03:20:26.068927+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:19:05.974214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:19:05.974214+0000","last_clean_scrub_stamp":"2024-09-22T03:19:05.974214+0000","objects_scrubbed":0,"log_size":631,"log_dups_size":0,"ondisk_log_size":631,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5177344,"num_objects":79,"num_object_clones":0,"num_object_copies":237,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":632,"num_write_kb":5056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.8","version":"144'679","reported_seq":718,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-22T03:20:25.595486+0000","last_change":"2024-09-22T03:20:15.033915+0000","last_active":"2024-09-22T03:20:25.595486+0000","last_peered":"2024-09-22T03:20:25.595486+0000","last_clean":"2024-09-22T03:20:25.595486+0000","last_became_active":"2024-09-22T03:20:15.023922+0000","last_became_peered":"2024-09-22T03:20:15.023922+0000","last_unstale":"2024-09-22T03:20:25.595486+0000","last_undegraded":"2024-09-22T03:20:25.595486+0000","last_fullsized":"2024-09-22T03:20:25.595486+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:19:05.974214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:19:05.974214+0000","last_clean_scrub_stamp":"2024-09-22T03:19:05.974214+0000","objects_scrubbed":0,"log_size":679,"log_dups_size":0,"ondisk_log_size":679,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5570560,"num_objects":85,"num_object_clones":0,"num_object_copies":255,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":680,"num_write_kb":5440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":202,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-22T03:20:18.976646+0000","last_change":"2024-09-22T03:20:18.978285+0000","last_active":"2024-09-22T03:20:18.976646+0000","last_peered":"2024-09-22T03:20:18.976646+0000","last_clean":"2024-09-22T03:20:18.976646+0000","last_became_active":"2024-09-22T03:20:15.050795+0000","last_became_peered":"2024-09-22T03:20:15.050795+0000","last_unstale":"2024-09-22T03:20:18.976646+0000","last_undegraded":"2024-09-22T03:20:18.976646+0000","last_fullsized":"2024-09-22T03:20:18.976646+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.9","version":"144'802","reported_seq":845,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-22T03:20:25.736492+0000","last_change":"2024-09-22T03:20:15.132169+0000","last_active":"2024-09-22T03:20:25.736492+0000","last_peered":"2024-09-22T03:20:25.736492+0000","last_clean":"2024-09-22T03:20:25.736492+0000","last_became_active":"2024-09-22T03:20:15.036460+0000","last_became_peered":"2024-09-22T03:20:15.036460+0000","last_unstale":"2024-09-22T03:20:25.736492+0000","last_undegraded":"2024-09-22T03:20:25.736492+0000","last_fullsized":"2024-09-22T03:20:25.736492+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:19:05.974214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:19:05.974214+0000","last_clean_scrub_stamp":"2024-09-22T03:19:05.974214+0000","objects_scrubbed":0,"log_size":802,"log_dups_size":0,"ondisk_log_size":802,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6578176,"num_objects":101,"num_object_clones":0,"num_object_copies":303,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":803,"num_write_kb":6424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":8192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":355,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-22T03:20:18.962519+0000","last_change":"2024-09-22T03:20:18.964214+0000","last_active":"2024-09-22T03:20:18.962519+0000","last_peered":"2024-09-22T03:20:18.962519+0000","last_clean":"2024-09-22T03:20:18.962519+0000","last_became_active":"2024-09-22T03:20:15.069435+0000","last_became_peered":"2024-09-22T03:20:15.069435+0000","last_unstale":"2024-09-22T03:20:18.962519+0000","last_undegraded":"2024-09-22T03:20:18.962519+0000","last_fullsized":"2024-09-22T03:20:18.962519+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.a","version":"144'799","reported_seq":837,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-22T03:20:26.683738+0000","last_change":"2024-09-22T03:20:15.047241+0000","last_active":"2024-09-22T03:20:26.683738+0000","last_peered":"2024-09-22T03:20:26.683738+0000","last_clean":"2024-09-22T03:20:26.683738+0000","last_became_active":"2024-09-22T03:20:15.042755+0000","last_became_peered":"2024-09-22T03:20:15.042755+0000","last_unstale":"2024-09-22T03:20:26.683738+0000","last_undegraded":"2024-09-22T03:20:26.683738+0000","last_fullsized":"2024-09-22T03:20:26.683738+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:19:05.974214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:19:05.974214+0000","last_clean_scrub_stamp":"2024-09-22T03:19:05.974214+0000","objects_scrubbed":0,"log_size":799,"log_dups_size":0,"ondisk_log_size":799,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6553600,"num_objects":100,"num_object_clones":0,"num_object_copies":300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":800,"num_write_kb":6400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":357,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-22T03:20:18.951930+0000","last_change":"2024-09-22T03:19:33.954680+0000","last_active":"2024-09-22T03:20:18.951930+0000","last_peered":"2024-09-22T03:20:18.951930+0000","last_clean":"2024-09-22T03:20:18.951930+0000","last_became_active":"2024-09-22T03:19:28.997839+0000","last_became_peered":"2024-09-22T03:19:28.997839+0000","last_unstale":"2024-09-22T03:20:18.951930+0000","last_undegraded":"2024-09-22T03:20:18.951930+0000","last_fullsized":"2024-09-22T03:20:18.951930+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.b","version":"144'695","reported_seq":733,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-22T03:20:28.057490+0000","last_change":"2024-09-22T03:20:15.084294+0000","last_active":"2024-09-22T03:20:28.057490+0000","last_peered":"2024-09-22T03:20:28.057490+0000","last_clean":"2024-09-22T03:20:28.057490+0000","last_became_active":"2024-09-22T03:20:15.075210+0000","last_became_peered":"2024-09-22T03:20:15.075210+0000","last_unstale":"2024-09-22T03:20:28.057490+0000","last_undegraded":"2024-09-22T03:20:28.057490+0000","last_fullsized":"2024-09-22T03:20:28.057490+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:19:05.974214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:19:05.974214+0000","last_clean_scrub_stamp":"2024-09-22T03:19:05.974214+0000","objects_scrubbed":0,"log_size":695,"log_dups_size":0,"ondisk_log_size":695,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5701632,"num_objects":87,"num_object_clones":0,"num_object_copies":261,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":696,"num_write_kb":5568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":232,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-22T03:20:18.976891+0000","last_change":"2024-09-22T03:20:18.977947+0000","last_active":"2024-09-22T03:20:18.976891+0000","last_peered":"2024-09-22T03:20:18.976891+0000","last_clean":"2024-09-22T03:20:18.976891+0000","last_became_active":"2024-09-22T03:20:15.042755+0000","last_became_peered":"2024-09-22T03:20:15.042755+0000","last_unstale":"2024-09-22T03:20:18.976891+0000","last_undegraded":"2024-09-22T03:20:18.976891+0000","last_fullsized":"2024-09-22T03:20:18.976891+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"12'97","reported_seq":475,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-22T03:20:18.960335+0000","last_change":"2024-09-22T03:20:15.058640+0000","last_active":"2024-09-22T03:20:18.960335+0000","last_peered":"2024-09-22T03:20:18.960335+0000","last_clean":"2024-09-22T03:20:18.960335+0000","last_became_active":"2024-09-22T03:20:15.055409+0000","last_became_peered":"2024-09-22T03:20:15.055409+0000","last_unstale":"2024-09-22T03:20:18.960335+0000","last_undegraded":"2024-09-22T03:20:18.960335+0000","last_fullsized":"2024-09-22T03:20:18.960335+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.c","version":"144'615","reported_seq":655,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-22T03:20:27.059839+0000","last_change":"2024-09-22T03:20:15.172278+0000","last_active":"2024-09-22T03:20:27.059839+0000","last_peered":"2024-09-22T03:20:27.059839+0000","last_clean":"2024-09-22T03:20:27.059839+0000","last_became_active":"2024-09-22T03:20:15.039842+0000","last_became_peered":"2024-09-22T03:20:15.039842+0000","last_unstale":"2024-09-22T03:20:27.059839+0000","last_undegraded":"2024-09-22T03:20:27.059839+0000","last_fullsized":"2024-09-22T03:20:27.059839+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:19:05.974214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:19:05.974214+0000","last_clean_scrub_stamp":"2024-09-22T03:19:05.974214+0000","objects_scrubbed":0,"log_size":615,"log_dups_size":0,"ondisk_log_size":615,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5046272,"num_objects":77,"num_object_clones":0,"num_object_copies":231,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":616,"num_write_kb":4928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":24576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":266,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-22T03:20:18.956818+0000","last_change":"2024-09-22T03:20:18.957911+0000","last_active":"2024-09-22T03:20:18.956818+0000","last_peered":"2024-09-22T03:20:18.956818+0000","last_clean":"2024-09-22T03:20:18.956818+0000","last_became_active":"2024-09-22T03:20:15.017488+0000","last_became_peered":"2024-09-22T03:20:15.017488+0000","last_unstale":"2024-09-22T03:20:18.956818+0000","last_undegraded":"2024-09-22T03:20:18.956818+0000","last_fullsized":"2024-09-22T03:20:18.956818+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.d","version":"144'663","reported_seq":706,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-22T03:20:25.369852+0000","last_change":"2024-09-22T03:20:15.174480+0000","last_active":"2024-09-22T03:20:25.369852+0000","last_peered":"2024-09-22T03:20:25.369852+0000","last_clean":"2024-09-22T03:20:25.369852+0000","last_became_active":"2024-09-22T03:20:15.059978+0000","last_became_peered":"2024-09-22T03:20:15.059978+0000","last_unstale":"2024-09-22T03:20:25.369852+0000","last_undegraded":"2024-09-22T03:20:25.369852+0000","last_fullsized":"2024-09-22T03:20:25.369852+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:19:05.974214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:19:05.974214+0000","last_clean_scrub_stamp":"2024-09-22T03:19:05.974214+0000","objects_scrubbed":0,"log_size":663,"log_dups_size":0,"ondisk_log_size":663,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5439488,"num_objects":83,"num_object_clones":0,"num_object_copies":249,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":664,"num_write_kb":5312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":49152,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":332,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-22T03:20:18.953159+0000","last_change":"2024-09-22T03:20:18.954496+0000","last_active":"2024-09-22T03:20:18.953159+0000","last_peered":"2024-09-22T03:20:18.953159+0000","last_clean":"2024-09-22T03:20:18.953159+0000","last_became_active":"2024-09-22T03:20:15.050168+0000","last_became_peered":"2024-09-22T03:20:15.050168+0000","last_unstale":"2024-09-22T03:20:18.953159+0000","last_undegraded":"2024-09-22T03:20:18.953159+0000","last_fullsized":"2024-09-22T03:20:18.953159+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.e","version":"144'775","reported_seq":813,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-22T03:20:28.000098+0000","last_change":"2024-09-22T03:20:15.072285+0000","last_active":"2024-09-22T03:20:28.000098+0000","last_peered":"2024-09-22T03:20:28.000098+0000","last_clean":"2024-09-22T03:20:28.000098+0000","last_became_active":"2024-09-22T03:20:15.066796+0000","last_became_peered":"2024-09-22T03:20:15.066796+0000","last_unstale":"2024-09-22T03:20:28.000098+0000","last_undegraded":"2024-09-22T03:20:28.000098+0000","last_fullsized":"2024-09-22T03:20:28.000098+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:19:05.974214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:19:05.974214+0000","last_clean_scrub_stamp":"2024-09-22T03:19:05.974214+0000","objects_scrubbed":0,"log_size":775,"log_dups_size":0,"ondisk_log_size":775,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6356992,"num_objects":97,"num_object_clones":0,"num_object_copies":291,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":776,"num_write_kb":6208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":341,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-22T03:20:18.955414+0000","last_change":"2024-09-22T03:19:33.964315+0000","last_active":"2024-09-22T03:20:18.955414+0000","last_peered":"2024-09-22T03:20:18.955414+0000","last_clean":"2024-09-22T03:20:18.955414+0000","last_became_active":"2024-09-22T03:19:29.013783+0000","last_became_peered":"2024-09-22T03:19:29.013783+0000","last_unstale":"2024-09-22T03:20:18.955414+0000","last_undegraded":"2024-09-22T03:20:18.955414+0000","last_fullsized":"2024-09-22T03:20:18.955414+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.f","version":"144'679","reported_seq":717,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-22T03:20:28.024382+0000","last_change":"2024-09-22T03:20:15.061145+0000","last_active":"2024-09-22T03:20:28.024382+0000","last_peered":"2024-09-22T03:20:28.024382+0000","last_clean":"2024-09-22T03:20:28.024382+0000","last_became_active":"2024-09-22T03:20:15.055489+0000","last_became_peered":"2024-09-22T03:20:15.055489+0000","last_unstale":"2024-09-22T03:20:28.024382+0000","last_undegraded":"2024-09-22T03:20:28.024382+0000","last_fullsized":"2024-09-22T03:20:28.024382+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:19:05.974214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:19:05.974214+0000","last_clean_scrub_stamp":"2024-09-22T03:19:05.974214+0000","objects_scrubbed":0,"log_size":679,"log_dups_size":0,"ondisk_log_size":679,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5570560,"num_objects":85,"num_object_clones":0,"num_object_copies":255,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":680,"num_write_kb":5440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":336,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-22T03:20:18.956520+0000","last_change":"2024-09-22T03:20:18.964527+0000","last_active":"2024-09-22T03:20:18.956520+0000","last_peered":"2024-09-22T03:20:18.956520+0000","last_clean":"2024-09-22T03:20:18.956520+0000","last_became_active":"2024-09-22T03:20:15.035697+0000","last_became_peered":"2024-09-22T03:20:15.035697+0000","last_unstale":"2024-09-22T03:20:18.956520+0000","last_undegraded":"2024-09-22T03:20:18.956520+0000","last_fullsized":"2024-09-22T03:20:18.956520+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":92872704,"num_objects":1418,"num_object_clones":0,"num_object_copies":4254,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11337,"num_write_kb":90696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":90112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":139354112,"data_stored":278765568,"data_compressed":14390684,"data_compressed_allocated":139354112,"data_compressed_original":278765568,"omap_allocated":0,"internal_metadata":0},"log_size":11321,"ondisk_log_size":11321,"up":48,"acting":48,"num_store_stats":4},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1180736,"data_compressed":59924,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":130,"seq":558345748574,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":65736,"kb_used_data":36472,"kb_used_omap":0,"kb_used_meta":29248,"kb_avail":94306104,"statfs":{"total":96636764160,"available":96569450496,"internally_reserved":0,"allocated":37347328,"data_stored":72758672,"data_compressed":3730832,"data_compressed_allocated":36134912,"data_compressed_original":72269824,"omap_allocated":0,"internal_metadata":29949952},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":100,"seq":429496729849,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":75644,"kb_used_data":29164,"kb_used_omap":0,"kb_used_meta":46464,"kb_avail":94296196,"statfs":{"total":96636764160,"available":96559304704,"internally_reserved":0,"allocated":29863936,"data_stored":58259376,"data_compressed":2981802,"data_compressed_allocated":28631040,"data_compressed_original":57753600,"omap_allocated":0,"internal_metadata":47579136},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":138,"seq":592705486907,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":62288,"kb_used_data":28736,"kb_used_omap":0,"kb_used_meta":33536,"kb_avail":94309552,"statfs":{"total":96636764160,"available":96572981248,"internally_reserved":0,"allocated":29425664,"data_stored":57382832,"data_compressed":2936515,"data_compressed_allocated":28192768,"data_compressed_original":56877056,"omap_allocated":0,"internal_metadata":34340864},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":142,"seq":609885356046,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":74992,"kb_used_data":44768,"kb_used_omap":0,"kb_used_meta":30208,"kb_avail":94296848,"statfs":{"total":96636764160,"available":96559972352,"internally_reserved":0,"allocated":45842432,"data_stored":90281885,"data_compressed":4634727,"data_compressed_allocated":44613632,"data_compressed_original":89776128,"omap_allocated":0,"internal_metadata":30932992},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29962,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29962,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":44765184,"data_stored":89587712,"data_compressed":4624646,"data_compressed_allocated":44765184,"data_compressed_original":89587712,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28745728,"data_stored":57491456,"data_compressed":2967888,"data_compressed_allocated":28745728,"data_compressed_original":57491456,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":29032448,"data_stored":58064896,"data_compressed":2997523,"data_compressed_allocated":29032448,"data_compressed_original":58064896,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":36810752,"data_stored":73621504,"data_compressed":3800627,"data_compressed_allocated":36810752,"data_compressed_original":73621504,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-22T03:20:30.075 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 82 16 12273 12257 1.16766 2.33594 0.0522665 0.107013 2024-09-22T03:20:31.075 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 83 16 12534 12518 1.17816 2.03906 0.078707 0.106027 2024-09-22T03:20:31.186 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-22T03:20:31.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:20:31.311+0000 7f6cdea2c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:20:31.320 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:20:31.318+0000 7f6cdea2c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:20:32.075 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 84 16 12829 12813 1.19157 2.30469 0.029875 0.104866 2024-09-22T03:20:33.075 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 85 16 13128 13112 1.20503 2.33594 0.082291 0.103687 2024-09-22T03:20:33.599 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-22T03:20:33.749 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:20:33.746+0000 7f1776d17640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:20:33.753 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:20:33.752+0000 7f1776d17640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:20:34.075 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 86 16 13442 13426 1.21954 2.45312 0.0545262 0.102459 2024-09-22T03:20:34.889 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [2, 3, 1, 0] 2024-09-22T03:20:34.889 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-22T03:20:34.889 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-09-22T03:20:34.890 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-09-22T03:20:35.040 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:20:35.029+0000 7ff2bbad7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:20:35.047 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:20:35.045+0000 7ff2bbad7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:20:35.069 INFO:teuthology.orchestra.run.smithi153.stderr:no valid command found; 10 closest matches: 2024-09-22T03:20:35.069 INFO:teuthology.orchestra.run.smithi153.stderr:config get 2024-09-22T03:20:35.069 INFO:teuthology.orchestra.run.smithi153.stderr:config help 2024-09-22T03:20:35.069 INFO:teuthology.orchestra.run.smithi153.stderr:config set ... 2024-09-22T03:20:35.069 INFO:teuthology.orchestra.run.smithi153.stderr:config show 2024-09-22T03:20:35.069 INFO:teuthology.orchestra.run.smithi153.stderr:deep_scrub 2024-09-22T03:20:35.069 INFO:teuthology.orchestra.run.smithi153.stderr:dump_historic_ops 2024-09-22T03:20:35.069 INFO:teuthology.orchestra.run.smithi153.stderr:dump_historic_slow_ops 2024-09-22T03:20:35.069 INFO:teuthology.orchestra.run.smithi153.stderr:dump_metrics [] 2024-09-22T03:20:35.069 INFO:teuthology.orchestra.run.smithi153.stderr:dump_ops_in_flight 2024-09-22T03:20:35.070 INFO:teuthology.orchestra.run.smithi153.stderr:dump_pgstate_history 2024-09-22T03:20:35.070 INFO:teuthology.orchestra.run.smithi153.stderr:Error EINVAL: invalid command 2024-09-22T03:20:35.072 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-22T03:20:35.073 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-22T03:20:35.073 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-09-22T03:20:35.076 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 87 16 13732 13716 1.23156 2.26562 0.0441332 0.101461 2024-09-22T03:20:35.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:20:35.202+0000 7f3e60ad7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:20:35.217 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:20:35.215+0000 7f3e60ad7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:20:35.236 INFO:teuthology.orchestra.run.smithi153.stderr:no valid command found; 10 closest matches: 2024-09-22T03:20:35.236 INFO:teuthology.orchestra.run.smithi153.stderr:config get 2024-09-22T03:20:35.236 INFO:teuthology.orchestra.run.smithi153.stderr:config help 2024-09-22T03:20:35.236 INFO:teuthology.orchestra.run.smithi153.stderr:config set ... 2024-09-22T03:20:35.236 INFO:teuthology.orchestra.run.smithi153.stderr:config show 2024-09-22T03:20:35.237 INFO:teuthology.orchestra.run.smithi153.stderr:deep_scrub 2024-09-22T03:20:35.237 INFO:teuthology.orchestra.run.smithi153.stderr:dump_historic_ops 2024-09-22T03:20:35.237 INFO:teuthology.orchestra.run.smithi153.stderr:dump_historic_slow_ops 2024-09-22T03:20:35.237 INFO:teuthology.orchestra.run.smithi153.stderr:dump_metrics [] 2024-09-22T03:20:35.237 INFO:teuthology.orchestra.run.smithi153.stderr:dump_ops_in_flight 2024-09-22T03:20:35.237 INFO:teuthology.orchestra.run.smithi153.stderr:dump_pgstate_history 2024-09-22T03:20:35.237 INFO:teuthology.orchestra.run.smithi153.stderr:Error EINVAL: invalid command 2024-09-22T03:20:35.240 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-22T03:20:35.240 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-22T03:20:35.240 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-09-22T03:20:35.384 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:20:35.378+0000 7fb753397640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:20:35.389 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:20:35.387+0000 7fb753397640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:20:35.405 INFO:teuthology.orchestra.run.smithi153.stderr:no valid command found; 10 closest matches: 2024-09-22T03:20:35.405 INFO:teuthology.orchestra.run.smithi153.stderr:config get 2024-09-22T03:20:35.405 INFO:teuthology.orchestra.run.smithi153.stderr:config help 2024-09-22T03:20:35.405 INFO:teuthology.orchestra.run.smithi153.stderr:config set ... 2024-09-22T03:20:35.405 INFO:teuthology.orchestra.run.smithi153.stderr:config show 2024-09-22T03:20:35.405 INFO:teuthology.orchestra.run.smithi153.stderr:deep_scrub 2024-09-22T03:20:35.405 INFO:teuthology.orchestra.run.smithi153.stderr:dump_historic_ops 2024-09-22T03:20:35.405 INFO:teuthology.orchestra.run.smithi153.stderr:dump_historic_slow_ops 2024-09-22T03:20:35.405 INFO:teuthology.orchestra.run.smithi153.stderr:dump_metrics [] 2024-09-22T03:20:35.405 INFO:teuthology.orchestra.run.smithi153.stderr:dump_ops_in_flight 2024-09-22T03:20:35.405 INFO:teuthology.orchestra.run.smithi153.stderr:dump_pgstate_history 2024-09-22T03:20:35.405 INFO:teuthology.orchestra.run.smithi153.stderr:Error EINVAL: invalid command 2024-09-22T03:20:35.408 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-22T03:20:35.409 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-22T03:20:35.409 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-09-22T03:20:35.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:20:35.549+0000 7faf4d517640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:20:35.562 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:20:35.560+0000 7faf4d517640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:20:35.628 INFO:teuthology.orchestra.run.smithi153.stderr:no valid command found; 10 closest matches: 2024-09-22T03:20:35.628 INFO:teuthology.orchestra.run.smithi153.stderr:config get 2024-09-22T03:20:35.628 INFO:teuthology.orchestra.run.smithi153.stderr:config help 2024-09-22T03:20:35.628 INFO:teuthology.orchestra.run.smithi153.stderr:config set ... 2024-09-22T03:20:35.628 INFO:teuthology.orchestra.run.smithi153.stderr:config show 2024-09-22T03:20:35.628 INFO:teuthology.orchestra.run.smithi153.stderr:deep_scrub 2024-09-22T03:20:35.628 INFO:teuthology.orchestra.run.smithi153.stderr:dump_historic_ops 2024-09-22T03:20:35.629 INFO:teuthology.orchestra.run.smithi153.stderr:dump_historic_slow_ops 2024-09-22T03:20:35.629 INFO:teuthology.orchestra.run.smithi153.stderr:dump_metrics [] 2024-09-22T03:20:35.629 INFO:teuthology.orchestra.run.smithi153.stderr:dump_ops_in_flight 2024-09-22T03:20:35.629 INFO:teuthology.orchestra.run.smithi153.stderr:dump_pgstate_history 2024-09-22T03:20:35.629 INFO:teuthology.orchestra.run.smithi153.stderr:Error EINVAL: invalid command 2024-09-22T03:20:35.631 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-22T03:20:35.632 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-22T03:20:35.863 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-22T03:20:36.008 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:20:36.004+0000 7f75fad57640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:20:36.012 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:20:36.010+0000 7f75fad57640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:20:36.075 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 88 16 14019 14003 1.24304 2.24219 0.0819636 0.100493 2024-09-22T03:20:37.075 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 89 16 14327 14311 1.25611 2.40625 0.0420971 0.0994823 2024-09-22T03:20:38.075 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 90 16 14610 14594 1.26671 2.21094 0.0277144 0.098627 2024-09-22T03:20:38.256 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-22T03:20:38.359 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:20:38.356+0000 7f98a1ab2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:20:38.362 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:20:38.360+0000 7f98a1ab2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:20:39.075 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:Total time run: 90.0531 2024-09-22T03:20:39.076 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:Total writes made: 14610 2024-09-22T03:20:39.076 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:Write size: 8192 2024-09-22T03:20:39.076 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:Object size: 65536 2024-09-22T03:20:39.076 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:Bandwidth (MB/sec): 1.26748 2024-09-22T03:20:39.076 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:Stddev Bandwidth: 1.25578 2024-09-22T03:20:39.076 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:Max bandwidth (MB/sec): 2.89844 2024-09-22T03:20:39.076 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:Min bandwidth (MB/sec): 0 2024-09-22T03:20:39.076 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:Average IOPS: 162 2024-09-22T03:20:39.076 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:Stddev IOPS: 160.752 2024-09-22T03:20:39.076 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:Max IOPS: 371 2024-09-22T03:20:39.076 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:Min IOPS: 0 2024-09-22T03:20:39.076 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:Average Latency(s): 0.0985935 2024-09-22T03:20:39.076 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:Stddev Latency(s): 1.04395 2024-09-22T03:20:39.076 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:Max latency(s): 23.6203 2024-09-22T03:20:39.077 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:Min latency(s): 0.00858012 2024-09-22T03:20:39.090 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:Cleaning up (deleting benchmark objects) 2024-09-22T03:20:40.428 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-22T03:20:40.569 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:20:40.565+0000 7f1b6db17640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:20:40.572 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:20:40.570+0000 7f1b6db17640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:20:40.633 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [2, 3, 1, 0] 2024-09-22T03:20:40.633 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-22T03:20:40.633 INFO:tasks.thrashosds.thrasher:Killing osd 3, live_osds are [2, 3, 1, 0] 2024-09-22T03:20:40.633 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-09-22T03:20:40.633 INFO:teuthology.orchestra.run:waiting for 300 2024-09-22T03:20:40.989 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~0s 2024-09-22T03:20:46.493 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~6s 2024-09-22T03:20:46.734 INFO:tasks.ceph.osd.3:Stopped 2024-09-22T03:20:51.736 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [3] live_osds: [2, 1, 0] 2024-09-22T03:20:51.736 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-22T03:20:51.736 INFO:tasks.thrashosds.thrasher:Reviving osd 3 2024-09-22T03:20:51.736 INFO:tasks.ceph.osd.3:Restarting daemon 2024-09-22T03:20:51.737 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2>> /var/log/ceph/ceph-osd.3.log 2024-09-22T03:20:51.739 INFO:tasks.ceph.osd.3:Started 2024-09-22T03:20:51.740 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-22T03:20:51.850 INFO:teuthology.orchestra.run.smithi196.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-09-22T03:20:51.853 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-22T03:20:51.854 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-09-22T03:20:52.099 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 has been restored 2024-09-22T03:20:56.856 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-22T03:20:56.951 INFO:teuthology.orchestra.run.smithi196.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-09-22T03:20:56.954 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-22T03:20:56.954 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-09-22T03:21:00.334 INFO:teuthology.orchestra.run.smithi153.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-09-22T03:21:00.338 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-22T03:21:00.338 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-22T03:21:01.959 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-22T03:21:02.067 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-09-22T03:21:02.068 INFO:teuthology.orchestra.run.smithi196.stdout: "ops_in_flight": [ 2024-09-22T03:21:02.068 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-22T03:21:02.068 INFO:teuthology.orchestra.run.smithi196.stdout: "type": "client_request", 2024-09-22T03:21:02.068 INFO:teuthology.orchestra.run.smithi196.stdout: "id": 74, 2024-09-22T03:21:02.068 INFO:teuthology.orchestra.run.smithi196.stdout: "detail": { 2024-09-22T03:21:02.068 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.068 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:21:02.068 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:21:01.408919+0000" 2024-09-22T03:21:02.068 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.068 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:21:02.068 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:21:02.068 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:21:01.408924+0000" 2024-09-22T03:21:02.068 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.069 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.069 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:21:02.069 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.069 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.069 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:21:02.069 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:21:02.069 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:21:01.409044+0000" 2024-09-22T03:21:02.069 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.069 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.069 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:21:02.069 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.069 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.069 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:21:02.069 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:21:02.070 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.070 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.070 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.070 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:21:02.070 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.070 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.070 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:21:02.070 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:21:02.070 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:21:01.409143+0000" 2024-09-22T03:21:02.070 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.070 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.070 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:21:02.070 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.070 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.071 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:21:02.071 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:21:02.071 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:21:01.409279+0000" 2024-09-22T03:21:02.071 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.071 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.071 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:21:02.071 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.071 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.071 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:21:02.071 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:21:02.071 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.071 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.071 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.071 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:21:02.072 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.072 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.072 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.072 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:21:02.072 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.072 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.072 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:21:02.072 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:21:02.072 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:21:01.409513+0000" 2024-09-22T03:21:02.072 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.072 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.072 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:21:02.072 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.073 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.073 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:21:02.073 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:21:02.073 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.073 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.073 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.073 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:21:02.073 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.073 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.073 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:21:02.073 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:21:02.073 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:21:01.409964+0000" 2024-09-22T03:21:02.073 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.073 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.074 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:21:02.074 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.074 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.074 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:21:02.074 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:21:02.074 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:21:01.410090+0000", 2024-09-22T03:21:02.074 INFO:teuthology.orchestra.run.smithi196.stdout: "blocker": { 2024-09-22T03:21:02.074 INFO:teuthology.orchestra.run.smithi196.stdout: "op_type": "PGActivationBlocker", 2024-09-22T03:21:02.074 INFO:teuthology.orchestra.run.smithi196.stdout: "detail": { 2024-09-22T03:21:02.074 INFO:teuthology.orchestra.run.smithi196.stdout: "pgid": "9.d" 2024-09-22T03:21:02.074 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:21:02.074 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:21:02.074 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.074 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.075 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:21:02.075 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.075 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.075 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:21:02.075 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:21:02.075 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.075 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.075 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.075 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:21:02.075 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.075 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.075 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:21:02.075 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:21:02.076 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.076 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.076 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.076 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:21:02.076 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.076 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.076 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:21:02.076 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:21:02.076 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.076 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.076 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.076 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:21:02.076 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.076 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.077 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:21:02.077 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:21:02.077 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.077 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.077 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.077 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:21:02.077 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.077 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.077 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:21:02.077 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:21:02.077 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.077 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.077 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.078 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:21:02.078 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.078 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.078 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:21:02.078 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:21:02.078 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.078 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.078 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.078 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:21:02.078 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.078 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.078 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:21:02.078 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:21:02.079 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.079 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.079 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.079 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:21:02.079 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.079 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.079 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:21:02.079 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:21:02.079 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.079 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.079 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.079 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:21:02.079 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.080 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.080 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:21:02.080 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:21:02.080 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.080 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.080 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.080 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:21:02.080 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.080 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.080 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.080 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:21:02.080 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.080 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:21:02.080 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:21:02.081 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.081 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-22T03:21:02.081 INFO:teuthology.orchestra.run.smithi196.stdout: "type": "client_request", 2024-09-22T03:21:02.081 INFO:teuthology.orchestra.run.smithi196.stdout: "id": 76, 2024-09-22T03:21:02.081 INFO:teuthology.orchestra.run.smithi196.stdout: "detail": { 2024-09-22T03:21:02.081 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.081 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:21:02.081 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:21:01.410178+0000" 2024-09-22T03:21:02.081 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.081 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:21:02.081 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:21:02.081 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:21:01.410182+0000" 2024-09-22T03:21:02.081 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.082 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.082 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:21:02.082 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.082 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.082 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:21:02.082 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:21:02.082 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:21:01.410370+0000" 2024-09-22T03:21:02.082 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.082 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.082 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:21:02.082 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.082 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.082 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:21:02.082 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:21:02.082 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.083 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.083 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.083 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:21:02.083 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.083 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.083 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:21:02.083 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:21:02.083 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:21:01.410557+0000" 2024-09-22T03:21:02.083 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.083 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.083 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:21:02.083 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.083 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.083 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:21:02.084 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:21:02.084 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:21:01.411000+0000" 2024-09-22T03:21:02.084 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.084 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.084 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:21:02.084 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.084 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.084 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:21:02.084 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:21:02.084 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.084 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.084 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.084 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:21:02.084 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.084 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.085 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.085 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:21:02.085 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.085 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.085 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:21:02.085 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:21:02.085 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:21:01.411259+0000" 2024-09-22T03:21:02.085 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.085 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.085 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:21:02.085 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.085 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.085 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:21:02.085 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:21:02.086 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.086 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.086 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.086 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:21:02.086 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.086 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.086 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:21:02.086 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:21:02.086 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:21:01.411454+0000", 2024-09-22T03:21:02.086 INFO:teuthology.orchestra.run.smithi196.stdout: "blocker": { 2024-09-22T03:21:02.086 INFO:teuthology.orchestra.run.smithi196.stdout: "op_type": "CommonPGPipeline:::wait_for_active", 2024-09-22T03:21:02.086 INFO:teuthology.orchestra.run.smithi196.stdout: "detail": { 2024-09-22T03:21:02.086 INFO:teuthology.orchestra.run.smithi196.stdout: "waiting": 1, 2024-09-22T03:21:02.086 INFO:teuthology.orchestra.run.smithi196.stdout: "held_by_operation_id": 74 2024-09-22T03:21:02.086 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:21:02.087 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:21:02.087 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.087 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.087 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:21:02.087 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.087 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.087 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:21:02.087 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:21:02.087 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.087 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.087 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.087 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:21:02.087 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.087 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.087 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:21:02.088 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:21:02.088 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.088 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.088 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.088 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:21:02.088 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.088 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.088 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:21:02.088 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:21:02.088 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.088 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.088 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.088 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:21:02.088 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.088 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.088 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:21:02.089 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:21:02.089 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.089 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.089 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.089 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:21:02.089 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.089 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.089 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:21:02.089 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:21:02.089 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.089 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.089 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.089 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:21:02.089 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.090 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.090 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:21:02.090 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:21:02.090 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.090 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.090 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.090 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:21:02.090 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.090 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.090 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:21:02.090 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:21:02.090 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.090 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.090 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.091 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:21:02.091 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.091 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.091 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:21:02.091 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:21:02.091 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.091 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.091 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.091 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:21:02.091 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.091 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.091 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:21:02.091 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:21:02.091 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.092 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.092 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.092 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:21:02.092 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.092 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.092 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:21:02.092 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:21:02.092 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.092 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.092 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.092 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:21:02.092 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.092 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.093 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.093 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:21:02.093 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.093 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:21:02.093 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:21:02.093 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.093 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-22T03:21:02.093 INFO:teuthology.orchestra.run.smithi196.stdout: "type": "client_request", 2024-09-22T03:21:02.093 INFO:teuthology.orchestra.run.smithi196.stdout: "id": 80, 2024-09-22T03:21:02.093 INFO:teuthology.orchestra.run.smithi196.stdout: "detail": { 2024-09-22T03:21:02.093 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.093 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:21:02.093 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:21:01.412735+0000" 2024-09-22T03:21:02.094 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.094 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:21:02.094 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:21:02.094 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:21:01.412739+0000" 2024-09-22T03:21:02.094 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.094 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.094 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:21:02.094 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.094 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.094 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:21:02.094 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:21:02.094 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:21:01.412827+0000" 2024-09-22T03:21:02.094 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.094 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.095 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:21:02.095 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.095 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.095 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:21:02.095 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:21:02.095 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.095 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.095 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.095 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:21:02.095 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.095 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.095 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:21:02.095 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:21:02.095 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:21:01.413124+0000" 2024-09-22T03:21:02.096 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.096 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.096 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:21:02.096 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.096 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.096 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:21:02.096 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:21:02.096 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:21:01.413317+0000" 2024-09-22T03:21:02.096 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.096 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.096 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:21:02.096 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.096 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.096 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:21:02.096 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:21:02.097 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.097 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.097 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.097 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:21:02.097 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.097 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.097 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.097 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:21:02.097 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.097 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.097 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:21:02.097 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:21:02.097 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:21:01.426257+0000", 2024-09-22T03:21:02.097 INFO:teuthology.orchestra.run.smithi196.stdout: "blocker": { 2024-09-22T03:21:02.098 INFO:teuthology.orchestra.run.smithi196.stdout: "op_type": "ClientRequest::PGPipeline::await_map", 2024-09-22T03:21:02.098 INFO:teuthology.orchestra.run.smithi196.stdout: "detail": { 2024-09-22T03:21:02.098 INFO:teuthology.orchestra.run.smithi196.stdout: "waiting": 1, 2024-09-22T03:21:02.098 INFO:teuthology.orchestra.run.smithi196.stdout: "held_by_operation_id": 76 2024-09-22T03:21:02.098 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:21:02.098 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:21:02.098 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.098 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.098 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:21:02.098 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.098 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.098 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:21:02.098 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:21:02.098 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.099 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.099 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.099 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:21:02.099 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.099 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.099 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:21:02.099 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:21:02.099 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.099 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.099 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.099 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:21:02.099 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.099 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.099 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:21:02.100 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:21:02.100 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.100 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.100 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.100 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:21:02.100 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.100 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.100 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:21:02.100 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:21:02.100 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.100 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.100 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.100 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:21:02.100 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.101 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.101 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:21:02.101 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:21:02.101 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.101 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.101 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.101 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:21:02.101 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.101 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.101 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:21:02.101 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:21:02.101 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.101 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.101 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.101 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:21:02.102 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.102 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.102 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:21:02.102 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:21:02.102 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.102 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.102 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.102 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:21:02.102 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.102 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.102 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:21:02.102 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:21:02.102 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.102 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.103 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.103 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:21:02.103 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.103 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.103 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:21:02.103 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:21:02.103 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.103 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.103 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.103 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:21:02.103 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.103 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.103 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:21:02.103 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:21:02.104 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.104 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.104 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.104 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:21:02.104 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.104 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.104 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:21:02.104 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:21:02.104 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.104 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.104 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.104 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:21:02.104 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.104 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.105 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:21:02.105 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:21:02.105 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.105 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.105 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.105 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:21:02.105 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.105 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.105 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.105 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:21:02.105 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.105 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:21:02.105 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:21:02.105 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.106 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-22T03:21:02.106 INFO:teuthology.orchestra.run.smithi196.stdout: "type": "client_request", 2024-09-22T03:21:02.106 INFO:teuthology.orchestra.run.smithi196.stdout: "id": 82, 2024-09-22T03:21:02.106 INFO:teuthology.orchestra.run.smithi196.stdout: "detail": { 2024-09-22T03:21:02.106 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.106 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:21:02.106 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:21:01.426919+0000" 2024-09-22T03:21:02.106 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.106 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:21:02.106 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:21:02.106 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:21:01.426925+0000" 2024-09-22T03:21:02.106 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.106 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.106 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:21:02.106 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.107 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.107 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:21:02.107 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:21:02.107 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:21:01.427518+0000" 2024-09-22T03:21:02.107 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.107 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.107 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:21:02.107 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.107 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.107 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:21:02.107 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:21:02.107 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.107 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.107 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.107 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:21:02.108 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.108 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.108 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:21:02.108 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:21:02.108 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:21:01.427634+0000" 2024-09-22T03:21:02.108 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.108 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.108 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:21:02.108 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.108 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.108 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:21:02.108 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:21:02.108 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:21:01.427803+0000", 2024-09-22T03:21:02.108 INFO:teuthology.orchestra.run.smithi196.stdout: "blocker": { 2024-09-22T03:21:02.108 INFO:teuthology.orchestra.run.smithi196.stdout: "op_type": "PerShardPipeline::create_or_wait_pg", 2024-09-22T03:21:02.108 INFO:teuthology.orchestra.run.smithi196.stdout: "detail": { 2024-09-22T03:21:02.109 INFO:teuthology.orchestra.run.smithi196.stdout: "waiting": 1, 2024-09-22T03:21:02.109 INFO:teuthology.orchestra.run.smithi196.stdout: "held_by_operation_id": 80 2024-09-22T03:21:02.109 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:21:02.109 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:21:02.109 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.109 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.109 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:21:02.109 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.109 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.109 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:21:02.109 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:21:02.109 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.109 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.109 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.109 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:21:02.110 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.110 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.110 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.110 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:21:02.110 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.110 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.110 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:21:02.110 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:21:02.110 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.110 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.110 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.110 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:21:02.110 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.111 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.111 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:21:02.111 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:21:02.111 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.111 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.111 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.111 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:21:02.111 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.111 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.111 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:21:02.111 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:21:02.111 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.111 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.111 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.112 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:21:02.112 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.112 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.112 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:21:02.112 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:21:02.112 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.112 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.112 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.112 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:21:02.112 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.112 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.112 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:21:02.112 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:21:02.112 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.112 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.113 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.113 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:21:02.113 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.113 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.113 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:21:02.113 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:21:02.113 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.113 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.113 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.113 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:21:02.113 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.113 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.113 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:21:02.113 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:21:02.114 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.114 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.114 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.114 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:21:02.114 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.114 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.114 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:21:02.114 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:21:02.114 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.114 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.114 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.114 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:21:02.114 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.114 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.115 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:21:02.115 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:21:02.115 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.115 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.115 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.115 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:21:02.115 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.115 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.115 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:21:02.115 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:21:02.115 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.115 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.115 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.115 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:21:02.116 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.116 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.116 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:21:02.116 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:21:02.116 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.116 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.116 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.116 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:21:02.116 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.116 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.116 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:21:02.116 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:21:02.116 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.116 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.116 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.117 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:21:02.117 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.117 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.117 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:21:02.117 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:21:02.117 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.117 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.117 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.117 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:21:02.117 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.117 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.117 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.117 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:21:02.117 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.118 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:21:02.118 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:21:02.118 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.118 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-22T03:21:02.118 INFO:teuthology.orchestra.run.smithi196.stdout: "type": "client_request", 2024-09-22T03:21:02.118 INFO:teuthology.orchestra.run.smithi196.stdout: "id": 84, 2024-09-22T03:21:02.118 INFO:teuthology.orchestra.run.smithi196.stdout: "detail": { 2024-09-22T03:21:02.118 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.118 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:21:02.118 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:21:01.428706+0000" 2024-09-22T03:21:02.118 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.118 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:21:02.118 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:21:02.118 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:21:01.428710+0000" 2024-09-22T03:21:02.118 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.119 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.119 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:21:02.119 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.119 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.119 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:21:02.119 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:21:02.119 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:21:01.428812+0000" 2024-09-22T03:21:02.119 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.119 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.119 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:21:02.119 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.119 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.119 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:21:02.119 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:21:02.120 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.120 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.120 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.120 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:21:02.120 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.120 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.120 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:21:02.120 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:21:02.120 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:21:01.428877+0000", 2024-09-22T03:21:02.120 INFO:teuthology.orchestra.run.smithi196.stdout: "blocker": { 2024-09-22T03:21:02.120 INFO:teuthology.orchestra.run.smithi196.stdout: "op_type": "ConnectionPipeline::get_pg_mapping", 2024-09-22T03:21:02.120 INFO:teuthology.orchestra.run.smithi196.stdout: "detail": { 2024-09-22T03:21:02.120 INFO:teuthology.orchestra.run.smithi196.stdout: "waiting": 1, 2024-09-22T03:21:02.120 INFO:teuthology.orchestra.run.smithi196.stdout: "held_by_operation_id": 82 2024-09-22T03:21:02.121 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:21:02.121 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:21:02.121 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.121 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.121 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:21:02.121 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.121 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.121 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:21:02.121 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:21:02.121 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.121 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.121 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.121 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:21:02.121 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.121 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.122 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:21:02.122 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:21:02.122 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.122 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.122 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.122 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:21:02.122 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.122 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.122 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.122 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:21:02.122 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.122 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.122 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:21:02.122 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:21:02.122 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.123 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.123 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.123 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:21:02.123 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.123 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.123 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:21:02.123 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:21:02.123 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.123 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.123 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.123 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:21:02.123 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.123 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.123 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:21:02.123 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:21:02.124 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.124 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.124 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.124 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:21:02.124 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.124 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.124 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:21:02.124 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:21:02.124 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.124 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.124 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.124 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:21:02.124 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.124 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.125 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:21:02.125 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:21:02.125 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.125 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.125 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.125 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:21:02.125 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.125 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.125 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:21:02.125 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:21:02.125 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.125 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.125 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.125 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:21:02.126 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.126 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.126 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:21:02.126 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:21:02.126 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.126 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.126 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.126 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:21:02.126 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.126 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.126 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:21:02.126 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:21:02.126 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.126 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.127 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.127 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:21:02.127 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.127 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.127 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:21:02.127 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:21:02.127 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.127 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.127 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.127 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:21:02.127 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.127 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.127 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:21:02.128 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:21:02.128 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.128 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.128 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.128 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:21:02.128 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.128 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.128 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:21:02.128 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:21:02.128 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.128 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.128 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.128 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:21:02.128 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.129 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.129 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:21:02.129 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:21:02.129 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.129 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.129 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.129 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:21:02.129 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.129 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.129 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:21:02.129 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:21:02.129 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.129 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.129 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.129 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:21:02.130 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.130 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.130 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.130 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:21:02.130 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.130 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:21:02.130 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:21:02.130 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.130 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-22T03:21:02.130 INFO:teuthology.orchestra.run.smithi196.stdout: "type": "client_request", 2024-09-22T03:21:02.130 INFO:teuthology.orchestra.run.smithi196.stdout: "id": 70, 2024-09-22T03:21:02.130 INFO:teuthology.orchestra.run.smithi196.stdout: "detail": { 2024-09-22T03:21:02.130 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.130 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:21:02.131 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:21:01.405021+0000" 2024-09-22T03:21:02.131 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.131 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:21:02.131 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:21:02.131 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:21:01.405026+0000" 2024-09-22T03:21:02.131 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.131 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.131 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:21:02.131 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.131 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.131 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:21:02.131 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:21:02.131 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:21:01.405357+0000" 2024-09-22T03:21:02.131 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.131 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.132 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:21:02.132 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.132 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.132 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:21:02.132 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:21:02.132 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.132 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.132 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.132 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:21:02.132 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.132 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.132 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:21:02.132 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:21:02.132 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:21:01.406203+0000" 2024-09-22T03:21:02.133 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.133 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.133 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:21:02.133 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.133 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.133 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:21:02.133 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:21:02.133 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:21:01.408132+0000" 2024-09-22T03:21:02.133 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.133 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.133 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:21:02.133 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.133 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.133 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:21:02.133 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:21:02.134 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.134 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.134 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.134 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:21:02.134 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.134 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.134 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.134 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:21:02.134 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.134 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.134 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:21:02.134 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:21:02.134 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:21:01.408620+0000" 2024-09-22T03:21:02.134 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.135 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.135 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:21:02.135 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.135 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.135 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:21:02.135 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:21:02.135 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.135 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.135 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.135 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:21:02.135 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.135 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.135 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:21:02.135 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:21:02.136 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:21:01.408803+0000" 2024-09-22T03:21:02.136 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.136 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.136 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:21:02.136 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.136 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.136 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:21:02.136 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:21:02.136 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:21:01.408907+0000", 2024-09-22T03:21:02.136 INFO:teuthology.orchestra.run.smithi196.stdout: "blocker": { 2024-09-22T03:21:02.136 INFO:teuthology.orchestra.run.smithi196.stdout: "op_type": "PGActivationBlocker", 2024-09-22T03:21:02.136 INFO:teuthology.orchestra.run.smithi196.stdout: "detail": { 2024-09-22T03:21:02.136 INFO:teuthology.orchestra.run.smithi196.stdout: "pgid": "9.9" 2024-09-22T03:21:02.136 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:21:02.137 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:21:02.137 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.137 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.137 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:21:02.137 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.137 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.137 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:21:02.137 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:21:02.137 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.137 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.137 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.137 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:21:02.137 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.137 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.137 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:21:02.138 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:21:02.138 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.138 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.138 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.138 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:21:02.138 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.138 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.138 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:21:02.138 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:21:02.138 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.138 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.138 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.138 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:21:02.138 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.139 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.139 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:21:02.139 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:21:02.139 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.139 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.139 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.139 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:21:02.139 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.139 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.139 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:21:02.139 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:21:02.139 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.139 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.139 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.140 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:21:02.140 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.140 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.140 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:21:02.140 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:21:02.140 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.140 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.140 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.140 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:21:02.140 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.140 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.140 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:21:02.140 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:21:02.140 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.140 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.141 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.141 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:21:02.141 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.141 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.141 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:21:02.141 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:21:02.141 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.141 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.141 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.141 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:21:02.141 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.141 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.141 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:21:02.141 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:21:02.141 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.142 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.142 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.142 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:21:02.142 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.142 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.142 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.142 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:21:02.142 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.142 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:21:02.142 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:21:02.142 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.142 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-22T03:21:02.142 INFO:teuthology.orchestra.run.smithi196.stdout: "type": "client_request", 2024-09-22T03:21:02.142 INFO:teuthology.orchestra.run.smithi196.stdout: "id": 61, 2024-09-22T03:21:02.142 INFO:teuthology.orchestra.run.smithi196.stdout: "detail": { 2024-09-22T03:21:02.143 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.143 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:21:02.143 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:21:01.401101+0000" 2024-09-22T03:21:02.143 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.143 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:21:02.143 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:21:02.143 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:21:01.401124+0000" 2024-09-22T03:21:02.143 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.143 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.143 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:21:02.143 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.143 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.143 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:21:02.143 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:21:02.144 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:21:01.401308+0000" 2024-09-22T03:21:02.144 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.144 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.144 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:21:02.144 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.144 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.144 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:21:02.144 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:21:02.144 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.144 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.144 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.144 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:21:02.144 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.144 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.144 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:21:02.145 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:21:02.145 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:21:01.401782+0000" 2024-09-22T03:21:02.145 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.145 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.145 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:21:02.145 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.145 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.145 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:21:02.145 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:21:02.145 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:21:01.402207+0000" 2024-09-22T03:21:02.145 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.145 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.145 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:21:02.145 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.145 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.146 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:21:02.146 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:21:02.146 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.146 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.146 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.146 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:21:02.146 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.146 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.146 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.146 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:21:02.146 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.146 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.146 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:21:02.146 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:21:02.147 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:21:01.402458+0000" 2024-09-22T03:21:02.147 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.147 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.147 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:21:02.147 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.147 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.147 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:21:02.147 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:21:02.147 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.147 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.147 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.147 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:21:02.147 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.147 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.147 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:21:02.148 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:21:02.148 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:21:01.402606+0000" 2024-09-22T03:21:02.148 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.148 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.148 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:21:02.148 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.148 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.148 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:21:02.148 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:21:02.148 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:21:01.402675+0000", 2024-09-22T03:21:02.148 INFO:teuthology.orchestra.run.smithi196.stdout: "blocker": { 2024-09-22T03:21:02.148 INFO:teuthology.orchestra.run.smithi196.stdout: "op_type": "PGActivationBlocker", 2024-09-22T03:21:02.148 INFO:teuthology.orchestra.run.smithi196.stdout: "detail": { 2024-09-22T03:21:02.148 INFO:teuthology.orchestra.run.smithi196.stdout: "pgid": "9.3" 2024-09-22T03:21:02.149 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:21:02.149 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:21:02.149 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.149 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.149 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:21:02.149 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.149 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.149 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:21:02.149 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:21:02.149 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.149 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.149 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.149 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:21:02.149 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.149 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.150 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:21:02.150 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:21:02.150 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.150 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.150 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.150 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:21:02.150 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.150 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.150 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:21:02.150 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:21:02.150 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.150 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.150 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.151 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:21:02.151 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.151 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.151 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:21:02.151 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:21:02.151 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.151 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.151 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.151 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:21:02.151 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.151 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.151 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:21:02.151 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:21:02.151 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.151 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.152 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.152 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:21:02.152 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.152 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.152 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:21:02.152 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:21:02.152 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.152 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.152 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.152 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:21:02.152 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.152 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.152 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:21:02.152 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:21:02.152 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.153 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.153 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.153 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:21:02.153 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.153 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.153 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:21:02.153 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:21:02.153 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.153 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.153 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.153 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:21:02.153 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.153 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.153 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:21:02.154 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:21:02.154 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.154 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.154 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.154 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:21:02.154 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.154 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.154 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.154 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:21:02.154 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.154 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:21:02.154 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:21:02.154 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.154 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-22T03:21:02.155 INFO:teuthology.orchestra.run.smithi196.stdout: "type": "client_request", 2024-09-22T03:21:02.155 INFO:teuthology.orchestra.run.smithi196.stdout: "id": 64, 2024-09-22T03:21:02.155 INFO:teuthology.orchestra.run.smithi196.stdout: "detail": { 2024-09-22T03:21:02.155 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.155 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:21:02.155 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:21:01.402806+0000" 2024-09-22T03:21:02.155 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.155 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:21:02.155 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:21:02.155 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:21:01.402811+0000" 2024-09-22T03:21:02.155 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.155 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.155 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:21:02.155 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.156 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.156 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:21:02.156 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:21:02.156 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:21:01.403019+0000" 2024-09-22T03:21:02.156 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.156 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.156 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:21:02.156 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.156 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.156 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:21:02.156 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:21:02.156 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.156 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.156 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.156 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:21:02.156 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.157 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.157 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:21:02.157 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:21:02.157 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:21:01.403402+0000" 2024-09-22T03:21:02.157 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.157 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.157 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:21:02.157 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.157 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.157 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:21:02.157 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:21:02.157 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:21:01.404266+0000" 2024-09-22T03:21:02.157 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.157 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.157 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:21:02.158 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.158 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.158 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:21:02.158 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:21:02.158 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.158 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.158 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.158 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:21:02.158 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.158 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.158 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.158 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:21:02.158 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.158 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.159 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:21:02.159 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:21:02.159 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:21:01.404480+0000" 2024-09-22T03:21:02.159 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.159 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.159 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:21:02.159 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.159 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.159 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:21:02.159 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:21:02.159 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.159 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.159 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.159 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:21:02.159 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.160 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.160 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:21:02.160 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:21:02.160 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:21:01.404578+0000" 2024-09-22T03:21:02.160 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.160 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.160 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:21:02.160 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.160 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.160 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:21:02.160 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:21:02.160 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:21:01.404642+0000", 2024-09-22T03:21:02.160 INFO:teuthology.orchestra.run.smithi196.stdout: "blocker": { 2024-09-22T03:21:02.160 INFO:teuthology.orchestra.run.smithi196.stdout: "op_type": "PGActivationBlocker", 2024-09-22T03:21:02.160 INFO:teuthology.orchestra.run.smithi196.stdout: "detail": { 2024-09-22T03:21:02.161 INFO:teuthology.orchestra.run.smithi196.stdout: "pgid": "9.8" 2024-09-22T03:21:02.161 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:21:02.161 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:21:02.161 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.161 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.161 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:21:02.161 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.161 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.161 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:21:02.161 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:21:02.161 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.161 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.161 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.161 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:21:02.162 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.162 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.162 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:21:02.162 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:21:02.162 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.162 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.162 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.162 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:21:02.162 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.162 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.162 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:21:02.162 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:21:02.162 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.162 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.162 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.163 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:21:02.163 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.163 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.163 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:21:02.163 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:21:02.163 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.163 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.163 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.163 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:21:02.163 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.163 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.163 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:21:02.163 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:21:02.163 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.163 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.164 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.164 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:21:02.164 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.164 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.164 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:21:02.164 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:21:02.164 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.164 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.164 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.164 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:21:02.164 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.164 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.164 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:21:02.164 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:21:02.164 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.165 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.165 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.165 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:21:02.165 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.165 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.165 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:21:02.165 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:21:02.165 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.165 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.165 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.165 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:21:02.165 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.165 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.165 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:21:02.166 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:21:02.166 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.166 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.166 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.166 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:21:02.166 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.166 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.166 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.166 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:21:02.166 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.166 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:21:02.166 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:21:02.166 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.166 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-22T03:21:02.166 INFO:teuthology.orchestra.run.smithi196.stdout: "type": "client_request", 2024-09-22T03:21:02.167 INFO:teuthology.orchestra.run.smithi196.stdout: "id": 72, 2024-09-22T03:21:02.167 INFO:teuthology.orchestra.run.smithi196.stdout: "detail": { 2024-09-22T03:21:02.167 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.167 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:21:02.167 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:21:01.407734+0000" 2024-09-22T03:21:02.167 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.167 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:21:02.167 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:21:02.167 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:21:01.407739+0000" 2024-09-22T03:21:02.167 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.167 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.167 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:21:02.167 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.167 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.168 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:21:02.168 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:21:02.168 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:21:01.408017+0000" 2024-09-22T03:21:02.168 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.168 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.168 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:21:02.168 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.168 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.168 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:21:02.168 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:21:02.168 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.168 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.168 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.168 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:21:02.169 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.169 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.169 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:21:02.169 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:21:02.169 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:21:01.408223+0000" 2024-09-22T03:21:02.169 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.169 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.169 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:21:02.169 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.169 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.169 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:21:02.169 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:21:02.169 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:21:01.408679+0000" 2024-09-22T03:21:02.169 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.170 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.170 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:21:02.170 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.170 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.170 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:21:02.170 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:21:02.170 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.170 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.170 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.170 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:21:02.170 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.170 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.170 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.170 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:21:02.171 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.171 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.171 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:21:02.171 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:21:02.171 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:21:01.408877+0000" 2024-09-22T03:21:02.171 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.171 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.171 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:21:02.171 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.171 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.171 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:21:02.171 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:21:02.171 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.171 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.172 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.172 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:21:02.172 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.172 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.172 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:21:02.172 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:21:02.172 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:21:01.408996+0000", 2024-09-22T03:21:02.172 INFO:teuthology.orchestra.run.smithi196.stdout: "blocker": { 2024-09-22T03:21:02.172 INFO:teuthology.orchestra.run.smithi196.stdout: "op_type": "CommonPGPipeline:::wait_for_active", 2024-09-22T03:21:02.172 INFO:teuthology.orchestra.run.smithi196.stdout: "detail": { 2024-09-22T03:21:02.172 INFO:teuthology.orchestra.run.smithi196.stdout: "waiting": 1, 2024-09-22T03:21:02.172 INFO:teuthology.orchestra.run.smithi196.stdout: "held_by_operation_id": 61 2024-09-22T03:21:02.172 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:21:02.172 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:21:02.172 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.172 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.173 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:21:02.173 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.173 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.173 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:21:02.173 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:21:02.173 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.173 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.173 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.173 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:21:02.173 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.173 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.173 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:21:02.173 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:21:02.173 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.173 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.174 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.174 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:21:02.174 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.174 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.174 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:21:02.174 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:21:02.174 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.174 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.174 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.174 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:21:02.174 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.174 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.174 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:21:02.174 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:21:02.174 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.175 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.175 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.175 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:21:02.175 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.175 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.175 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:21:02.175 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:21:02.175 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.175 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.175 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.175 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:21:02.175 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.175 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.175 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:21:02.176 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:21:02.176 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.176 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.176 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.176 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:21:02.176 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.176 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.176 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:21:02.176 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:21:02.176 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.176 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.176 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.176 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:21:02.176 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.177 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.177 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:21:02.177 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:21:02.177 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.177 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.177 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.177 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:21:02.177 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.177 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.177 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:21:02.177 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:21:02.177 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.177 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.177 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.178 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:21:02.178 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.178 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.178 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:21:02.178 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:21:02.178 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.178 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.178 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.178 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:21:02.178 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.178 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.178 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.178 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:21:02.178 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.179 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:21:02.179 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:21:02.179 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.179 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-22T03:21:02.179 INFO:teuthology.orchestra.run.smithi196.stdout: "type": "client_request", 2024-09-22T03:21:02.179 INFO:teuthology.orchestra.run.smithi196.stdout: "id": 77, 2024-09-22T03:21:02.179 INFO:teuthology.orchestra.run.smithi196.stdout: "detail": { 2024-09-22T03:21:02.179 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.179 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:21:02.179 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:21:01.411643+0000" 2024-09-22T03:21:02.179 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.179 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:21:02.179 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:21:02.179 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:21:01.411647+0000" 2024-09-22T03:21:02.180 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.180 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.180 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:21:02.180 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.180 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.180 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:21:02.180 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:21:02.180 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:21:01.411858+0000" 2024-09-22T03:21:02.180 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.180 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.180 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:21:02.180 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.180 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.181 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:21:02.181 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:21:02.181 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.181 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.181 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.181 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:21:02.181 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.181 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.181 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:21:02.181 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:21:02.181 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:21:01.411954+0000" 2024-09-22T03:21:02.181 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.181 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.181 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:21:02.181 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.182 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.182 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:21:02.182 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:21:02.182 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:21:01.412145+0000" 2024-09-22T03:21:02.182 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.182 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.182 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:21:02.182 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.182 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.182 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:21:02.182 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:21:02.182 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.182 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.182 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.183 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:21:02.183 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.183 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.183 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.183 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:21:02.183 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.183 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.183 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:21:02.183 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:21:02.183 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:21:01.412368+0000", 2024-09-22T03:21:02.183 INFO:teuthology.orchestra.run.smithi196.stdout: "blocker": { 2024-09-22T03:21:02.183 INFO:teuthology.orchestra.run.smithi196.stdout: "op_type": "ClientRequest::PGPipeline::await_map", 2024-09-22T03:21:02.183 INFO:teuthology.orchestra.run.smithi196.stdout: "detail": { 2024-09-22T03:21:02.183 INFO:teuthology.orchestra.run.smithi196.stdout: "waiting": 1, 2024-09-22T03:21:02.184 INFO:teuthology.orchestra.run.smithi196.stdout: "held_by_operation_id": 72 2024-09-22T03:21:02.184 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:21:02.184 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:21:02.184 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.184 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.184 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:21:02.184 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.184 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.184 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:21:02.184 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:21:02.184 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.184 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.184 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.184 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:21:02.184 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.185 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.185 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:21:02.185 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:21:02.185 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.185 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.185 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.185 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:21:02.185 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.185 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.185 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:21:02.185 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:21:02.185 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.185 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.185 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.185 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:21:02.186 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.186 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.186 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:21:02.186 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:21:02.186 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.186 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.186 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.186 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:21:02.186 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.186 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.186 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:21:02.186 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:21:02.186 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.186 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.186 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.187 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:21:02.187 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.187 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.187 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:21:02.187 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:21:02.187 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.187 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.187 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.187 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:21:02.187 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.187 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.187 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:21:02.187 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:21:02.187 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.187 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.188 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.188 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:21:02.188 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.188 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.188 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:21:02.188 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:21:02.188 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.188 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.188 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.188 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:21:02.188 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.188 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.188 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:21:02.188 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:21:02.188 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.189 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.189 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.189 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:21:02.189 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.189 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.189 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:21:02.189 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:21:02.189 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.189 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.189 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.189 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:21:02.189 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.189 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.189 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:21:02.190 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:21:02.190 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.190 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.190 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.190 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:21:02.190 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.190 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.190 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:21:02.190 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:21:02.190 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.190 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.190 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.190 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:21:02.190 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.190 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:21:02.190 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:21:02.191 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:21:02.191 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:21:02.191 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:21:02.191 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:21:02.191 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:21:02.191 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-22T03:21:02.191 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-09-22T03:21:02.340 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-22T03:21:02.475 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:21:02.471+0000 7f7b0bb57640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:21:02.480 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:21:02.478+0000 7f7b0bb57640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:21:04.640 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-22T03:21:04.775 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:21:04.771+0000 7f8a2bd17640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:21:04.781 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:21:04.779+0000 7f8a2bd17640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:21:06.476 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:Removed 1827 objects 2024-09-22T03:21:06.476 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:Clean up completed and total clean up time :27.3857 2024-09-22T03:21:06.982 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-22T03:21:07.086 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:21:07.083+0000 7f330fd17640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:21:07.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:21:07.088+0000 7f330fd17640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:21:07.192 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [2, 1, 0, 3] 2024-09-22T03:21:07.192 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-22T03:21:07.193 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-22T03:21:07.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:21:07.293+0000 7fca01c63640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:21:07.301 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:21:07.299+0000 7fca01c63640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:21:07.479 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-22T03:21:07.479 INFO:teuthology.orchestra.run.smithi153.stderr:dumped all 2024-09-22T03:21:07.492 INFO:teuthology.orchestra.run.smithi153.stdout:{"pg_ready":true,"pg_map":{"version":533,"stamp":"2024-09-22T03:21:06.273081+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":64504395,"num_objects":981,"num_object_clones":0,"num_object_copies":2943,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":104,"num_read_kb":92,"num_write":15601,"num_write_kb":118777,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":90112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":15550,"ondisk_log_size":15550,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":282632,"kb_used_data":100232,"kb_used_omap":0,"kb_used_meta":182272,"kb_avail":377204728,"statfs":{"total":386547056640,"available":386257641472,"internally_reserved":0,"allocated":102637568,"data_stored":198847162,"data_compressed":10162955,"data_compressed_allocated":97636352,"data_compressed_original":196804608,"omap_allocated":0,"internal_metadata":186646528},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-28770280,"num_objects":-438,"num_object_clones":0,"num_object_copies":-1314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":638,"num_write_kb":1409,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.943989"},"pg_stats":[{"pgid":"9.7","version":"147'1029","reported_seq":1077,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-09-22T03:21:03.059602+0000","last_change":"2024-09-22T03:21:02.471991+0000","last_active":"2024-09-22T03:21:03.059602+0000","last_peered":"2024-09-22T03:21:03.059602+0000","last_clean":"2024-09-22T03:21:03.059602+0000","last_became_active":"2024-09-22T03:21:02.466994+0000","last_became_peered":"2024-09-22T03:21:02.466994+0000","last_unstale":"2024-09-22T03:21:03.059602+0000","last_undegraded":"2024-09-22T03:21:03.059602+0000","last_fullsized":"2024-09-22T03:21:03.059602+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:19:05.974214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:19:05.974214+0000","last_clean_scrub_stamp":"2024-09-22T03:19:05.974214+0000","objects_scrubbed":0,"log_size":1029,"log_dups_size":0,"ondisk_log_size":1029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5046272,"num_objects":77,"num_object_clones":0,"num_object_copies":231,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1030,"num_write_kb":7872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.6","version":"147'971","reported_seq":1015,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-09-22T03:21:02.655428+0000","last_change":"2024-09-22T03:21:02.407961+0000","last_active":"2024-09-22T03:21:02.655428+0000","last_peered":"2024-09-22T03:21:02.655428+0000","last_clean":"2024-09-22T03:21:02.655428+0000","last_became_active":"2024-09-22T03:21:02.399331+0000","last_became_peered":"2024-09-22T03:21:02.399331+0000","last_unstale":"2024-09-22T03:21:02.655428+0000","last_undegraded":"2024-09-22T03:21:02.655428+0000","last_fullsized":"2024-09-22T03:21:02.655428+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:19:05.974214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:19:05.974214+0000","last_clean_scrub_stamp":"2024-09-22T03:19:05.974214+0000","objects_scrubbed":0,"log_size":971,"log_dups_size":0,"ondisk_log_size":971,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5308416,"num_objects":81,"num_object_clones":0,"num_object_copies":243,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":972,"num_write_kb":7488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.5","version":"144'907","reported_seq":947,"reported_epoch":146,"state":"peering","last_fresh":"2024-09-22T03:21:01.358939+0000","last_change":"2024-09-22T03:21:00.363362+0000","last_active":"2024-09-22T03:21:00.363362+0000","last_peered":"2024-09-22T03:20:40.667959+0000","last_clean":"2024-09-22T03:20:40.667959+0000","last_became_active":"2024-09-22T03:20:15.042430+0000","last_became_peered":"2024-09-22T03:20:15.042430+0000","last_unstale":"2024-09-22T03:21:01.358939+0000","last_undegraded":"2024-09-22T03:21:01.358939+0000","last_fullsized":"2024-09-22T03:21:01.358939+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:19:05.974214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:19:05.974214+0000","last_clean_scrub_stamp":"2024-09-22T03:19:05.974214+0000","objects_scrubbed":0,"log_size":907,"log_dups_size":0,"ondisk_log_size":907,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4849664,"num_objects":74,"num_object_clones":0,"num_object_copies":222,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":907,"num_write_kb":6976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.4","version":"144'795","reported_seq":834,"reported_epoch":146,"state":"peering","last_fresh":"2024-09-22T03:21:00.341724+0000","last_change":"2024-09-22T03:21:00.341724+0000","last_active":"2024-09-22T03:21:00.341724+0000","last_peered":"2024-09-22T03:20:40.463061+0000","last_clean":"2024-09-22T03:20:40.463061+0000","last_became_active":"2024-09-22T03:20:15.025087+0000","last_became_peered":"2024-09-22T03:20:15.025087+0000","last_unstale":"2024-09-22T03:21:00.341724+0000","last_undegraded":"2024-09-22T03:21:00.341724+0000","last_fullsized":"2024-09-22T03:21:00.341724+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:19:05.974214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:19:05.974214+0000","last_clean_scrub_stamp":"2024-09-22T03:19:05.974214+0000","objects_scrubbed":0,"log_size":795,"log_dups_size":0,"ondisk_log_size":795,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4521984,"num_objects":69,"num_object_clones":0,"num_object_copies":207,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":795,"num_write_kb":6144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,3],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3","version":"148'1128","reported_seq":1178,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-22T03:21:05.308480+0000","last_change":"2024-09-22T03:21:02.490259+0000","last_active":"2024-09-22T03:21:05.308480+0000","last_peered":"2024-09-22T03:21:05.308480+0000","last_clean":"2024-09-22T03:21:05.308480+0000","last_became_active":"2024-09-22T03:21:02.479223+0000","last_became_peered":"2024-09-22T03:21:02.479223+0000","last_unstale":"2024-09-22T03:21:05.308480+0000","last_undegraded":"2024-09-22T03:21:05.308480+0000","last_fullsized":"2024-09-22T03:21:05.308480+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:19:05.974214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:19:05.974214+0000","last_clean_scrub_stamp":"2024-09-22T03:19:05.974214+0000","objects_scrubbed":0,"log_size":1128,"log_dups_size":0,"ondisk_log_size":1128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1507328,"num_objects":23,"num_object_clones":0,"num_object_copies":69,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1129,"num_write_kb":8192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2","version":"147'817","reported_seq":861,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-09-22T03:21:02.579598+0000","last_change":"2024-09-22T03:20:15.028458+0000","last_active":"2024-09-22T03:21:02.579598+0000","last_peered":"2024-09-22T03:21:02.579598+0000","last_clean":"2024-09-22T03:21:02.579598+0000","last_became_active":"2024-09-22T03:20:15.024384+0000","last_became_peered":"2024-09-22T03:20:15.024384+0000","last_unstale":"2024-09-22T03:21:02.579598+0000","last_undegraded":"2024-09-22T03:21:02.579598+0000","last_fullsized":"2024-09-22T03:21:02.579598+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:19:05.974214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:19:05.974214+0000","last_clean_scrub_stamp":"2024-09-22T03:19:05.974214+0000","objects_scrubbed":0,"log_size":817,"log_dups_size":0,"ondisk_log_size":817,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":64,"num_object_clones":0,"num_object_copies":192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":818,"num_write_kb":6272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"147'1094","reported_seq":1152,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-09-22T03:21:03.050090+0000","last_change":"2024-09-22T03:21:02.618595+0000","last_active":"2024-09-22T03:21:03.050090+0000","last_peered":"2024-09-22T03:21:03.050090+0000","last_clean":"2024-09-22T03:21:03.050090+0000","last_became_active":"2024-09-22T03:21:02.463305+0000","last_became_peered":"2024-09-22T03:21:02.463305+0000","last_unstale":"2024-09-22T03:21:03.050090+0000","last_undegraded":"2024-09-22T03:21:03.050090+0000","last_fullsized":"2024-09-22T03:21:03.050090+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:19:05.974214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:19:05.974214+0000","last_clean_scrub_stamp":"2024-09-22T03:19:05.974214+0000","objects_scrubbed":0,"log_size":1094,"log_dups_size":0,"ondisk_log_size":1094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4915200,"num_objects":75,"num_object_clones":0,"num_object_copies":225,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1095,"num_write_kb":8320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":8192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.0","version":"144'868","reported_seq":912,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-22T03:21:01.350398+0000","last_change":"2024-09-22T03:20:15.012888+0000","last_active":"2024-09-22T03:21:01.350398+0000","last_peered":"2024-09-22T03:21:01.350398+0000","last_clean":"2024-09-22T03:21:01.350398+0000","last_became_active":"2024-09-22T03:20:15.005060+0000","last_became_peered":"2024-09-22T03:20:15.005060+0000","last_unstale":"2024-09-22T03:21:01.350398+0000","last_undegraded":"2024-09-22T03:21:01.350398+0000","last_fullsized":"2024-09-22T03:21:01.350398+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:19:05.974214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:19:05.974214+0000","last_clean_scrub_stamp":"2024-09-22T03:19:05.974214+0000","objects_scrubbed":0,"log_size":868,"log_dups_size":0,"ondisk_log_size":868,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4456448,"num_objects":68,"num_object_clones":0,"num_object_copies":204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":868,"num_write_kb":6656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.8","version":"148'961","reported_seq":1012,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-22T03:21:05.362922+0000","last_change":"2024-09-22T03:21:02.481834+0000","last_active":"2024-09-22T03:21:05.362922+0000","last_peered":"2024-09-22T03:21:05.362922+0000","last_clean":"2024-09-22T03:21:05.362922+0000","last_became_active":"2024-09-22T03:21:02.475503+0000","last_became_peered":"2024-09-22T03:21:02.475503+0000","last_unstale":"2024-09-22T03:21:05.362922+0000","last_undegraded":"2024-09-22T03:21:05.362922+0000","last_fullsized":"2024-09-22T03:21:05.362922+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:19:05.974214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:19:05.974214+0000","last_clean_scrub_stamp":"2024-09-22T03:19:05.974214+0000","objects_scrubbed":0,"log_size":961,"log_dups_size":0,"ondisk_log_size":961,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1245184,"num_objects":19,"num_object_clones":0,"num_object_copies":57,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":962,"num_write_kb":6976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":211,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-22T03:21:03.982405+0000","last_change":"2024-09-22T03:21:03.985334+0000","last_active":"2024-09-22T03:21:03.982405+0000","last_peered":"2024-09-22T03:21:03.982405+0000","last_clean":"2024-09-22T03:21:03.982405+0000","last_became_active":"2024-09-22T03:21:02.491715+0000","last_became_peered":"2024-09-22T03:21:02.491715+0000","last_unstale":"2024-09-22T03:21:03.982405+0000","last_undegraded":"2024-09-22T03:21:03.982405+0000","last_fullsized":"2024-09-22T03:21:03.982405+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.9","version":"148'1089","reported_seq":1144,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-22T03:21:05.251054+0000","last_change":"2024-09-22T03:21:02.447783+0000","last_active":"2024-09-22T03:21:05.251054+0000","last_peered":"2024-09-22T03:21:05.251054+0000","last_clean":"2024-09-22T03:21:05.251054+0000","last_became_active":"2024-09-22T03:21:02.443041+0000","last_became_peered":"2024-09-22T03:21:02.443041+0000","last_unstale":"2024-09-22T03:21:05.251054+0000","last_undegraded":"2024-09-22T03:21:05.251054+0000","last_fullsized":"2024-09-22T03:21:05.251054+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:19:05.974214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:19:05.974214+0000","last_clean_scrub_stamp":"2024-09-22T03:19:05.974214+0000","objects_scrubbed":0,"log_size":1089,"log_dups_size":0,"ondisk_log_size":1089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1114112,"num_objects":17,"num_object_clones":0,"num_object_copies":51,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1090,"num_write_kb":7872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":8192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":364,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-09-22T03:21:02.379296+0000","last_change":"2024-09-22T03:20:18.964214+0000","last_active":"2024-09-22T03:21:02.379296+0000","last_peered":"2024-09-22T03:21:02.379296+0000","last_clean":"2024-09-22T03:21:02.379296+0000","last_became_active":"2024-09-22T03:20:15.069435+0000","last_became_peered":"2024-09-22T03:20:15.069435+0000","last_unstale":"2024-09-22T03:21:02.379296+0000","last_undegraded":"2024-09-22T03:21:02.379296+0000","last_fullsized":"2024-09-22T03:21:02.379296+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.a","version":"144'1041","reported_seq":1080,"reported_epoch":146,"state":"peering","last_fresh":"2024-09-22T03:21:00.355268+0000","last_change":"2024-09-22T03:21:00.355268+0000","last_active":"2024-09-22T03:21:00.355268+0000","last_peered":"2024-09-22T03:20:40.504147+0000","last_clean":"2024-09-22T03:20:40.504147+0000","last_became_active":"2024-09-22T03:20:15.042755+0000","last_became_peered":"2024-09-22T03:20:15.042755+0000","last_unstale":"2024-09-22T03:21:00.355268+0000","last_undegraded":"2024-09-22T03:21:00.355268+0000","last_fullsized":"2024-09-22T03:21:00.355268+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:19:05.974214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:19:05.974214+0000","last_clean_scrub_stamp":"2024-09-22T03:19:05.974214+0000","objects_scrubbed":0,"log_size":1041,"log_dups_size":0,"ondisk_log_size":1041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6094848,"num_objects":93,"num_object_clones":0,"num_object_copies":279,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1041,"num_write_kb":8064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,3],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":359,"reported_epoch":146,"state":"peering","last_fresh":"2024-09-22T03:21:00.359714+0000","last_change":"2024-09-22T03:21:00.359714+0000","last_active":"2024-09-22T03:21:00.359714+0000","last_peered":"2024-09-22T03:20:18.953493+0000","last_clean":"2024-09-22T03:20:18.953493+0000","last_became_active":"2024-09-22T03:19:28.997839+0000","last_became_peered":"2024-09-22T03:19:28.997839+0000","last_unstale":"2024-09-22T03:21:00.359714+0000","last_undegraded":"2024-09-22T03:21:00.359714+0000","last_fullsized":"2024-09-22T03:21:00.359714+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,3],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.b","version":"147'928","reported_seq":982,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-09-22T03:21:03.066507+0000","last_change":"2024-09-22T03:21:02.627770+0000","last_active":"2024-09-22T03:21:03.066507+0000","last_peered":"2024-09-22T03:21:03.066507+0000","last_clean":"2024-09-22T03:21:03.066507+0000","last_became_active":"2024-09-22T03:21:02.484040+0000","last_became_peered":"2024-09-22T03:21:02.484040+0000","last_unstale":"2024-09-22T03:21:03.066507+0000","last_undegraded":"2024-09-22T03:21:03.066507+0000","last_fullsized":"2024-09-22T03:21:03.066507+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:19:05.974214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:19:05.974214+0000","last_clean_scrub_stamp":"2024-09-22T03:19:05.974214+0000","objects_scrubbed":0,"log_size":928,"log_dups_size":0,"ondisk_log_size":928,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4587520,"num_objects":70,"num_object_clones":0,"num_object_copies":210,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":929,"num_write_kb":7104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":241,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-22T03:21:03.983292+0000","last_change":"2024-09-22T03:21:03.989872+0000","last_active":"2024-09-22T03:21:03.983292+0000","last_peered":"2024-09-22T03:21:03.983292+0000","last_clean":"2024-09-22T03:21:03.983292+0000","last_became_active":"2024-09-22T03:21:02.455063+0000","last_became_peered":"2024-09-22T03:21:02.455063+0000","last_unstale":"2024-09-22T03:21:03.983292+0000","last_undegraded":"2024-09-22T03:21:03.983292+0000","last_fullsized":"2024-09-22T03:21:03.983292+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"12'97","reported_seq":484,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-09-22T03:21:02.348676+0000","last_change":"2024-09-22T03:20:15.058640+0000","last_active":"2024-09-22T03:21:02.348676+0000","last_peered":"2024-09-22T03:21:02.348676+0000","last_clean":"2024-09-22T03:21:02.348676+0000","last_became_active":"2024-09-22T03:20:15.055409+0000","last_became_peered":"2024-09-22T03:20:15.055409+0000","last_unstale":"2024-09-22T03:21:02.348676+0000","last_undegraded":"2024-09-22T03:21:02.348676+0000","last_fullsized":"2024-09-22T03:21:02.348676+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.c","version":"144'875","reported_seq":925,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-09-22T03:21:02.430864+0000","last_change":"2024-09-22T03:21:02.430864+0000","last_active":"2024-09-22T03:21:02.430864+0000","last_peered":"2024-09-22T03:21:02.430864+0000","last_clean":"2024-09-22T03:21:02.430864+0000","last_became_active":"2024-09-22T03:21:02.422738+0000","last_became_peered":"2024-09-22T03:21:02.422738+0000","last_unstale":"2024-09-22T03:21:02.430864+0000","last_undegraded":"2024-09-22T03:21:02.430864+0000","last_fullsized":"2024-09-22T03:21:02.430864+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:19:05.974214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:19:05.974214+0000","last_clean_scrub_stamp":"2024-09-22T03:19:05.974214+0000","objects_scrubbed":0,"log_size":875,"log_dups_size":0,"ondisk_log_size":875,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5111808,"num_objects":78,"num_object_clones":0,"num_object_copies":234,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":876,"num_write_kb":6784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":24576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":275,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-09-22T03:21:02.350897+0000","last_change":"2024-09-22T03:20:18.957911+0000","last_active":"2024-09-22T03:21:02.350897+0000","last_peered":"2024-09-22T03:21:02.350897+0000","last_clean":"2024-09-22T03:21:02.350897+0000","last_became_active":"2024-09-22T03:20:15.017488+0000","last_became_peered":"2024-09-22T03:20:15.017488+0000","last_unstale":"2024-09-22T03:21:02.350897+0000","last_undegraded":"2024-09-22T03:21:02.350897+0000","last_fullsized":"2024-09-22T03:21:02.350897+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.d","version":"148'1027","reported_seq":1083,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-22T03:21:05.345614+0000","last_change":"2024-09-22T03:21:02.449465+0000","last_active":"2024-09-22T03:21:05.345614+0000","last_peered":"2024-09-22T03:21:05.345614+0000","last_clean":"2024-09-22T03:21:05.345614+0000","last_became_active":"2024-09-22T03:21:02.440555+0000","last_became_peered":"2024-09-22T03:21:02.440555+0000","last_unstale":"2024-09-22T03:21:05.345614+0000","last_undegraded":"2024-09-22T03:21:05.345614+0000","last_fullsized":"2024-09-22T03:21:05.345614+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:19:05.974214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:19:05.974214+0000","last_clean_scrub_stamp":"2024-09-22T03:19:05.974214+0000","objects_scrubbed":0,"log_size":1027,"log_dups_size":0,"ondisk_log_size":1027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1703960,"num_objects":27,"num_object_clones":0,"num_object_copies":81,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1028,"num_write_kb":7489,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":49152,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":334,"reported_epoch":146,"state":"peering","last_fresh":"2024-09-22T03:21:00.352758+0000","last_change":"2024-09-22T03:21:00.352758+0000","last_active":"2024-09-22T03:21:00.352758+0000","last_peered":"2024-09-22T03:20:18.954496+0000","last_clean":"2024-09-22T03:20:18.954496+0000","last_became_active":"2024-09-22T03:20:15.050168+0000","last_became_peered":"2024-09-22T03:20:15.050168+0000","last_unstale":"2024-09-22T03:21:00.352758+0000","last_undegraded":"2024-09-22T03:21:00.352758+0000","last_fullsized":"2024-09-22T03:21:00.352758+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,3],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.e","version":"147'995","reported_seq":1042,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-09-22T03:21:03.089931+0000","last_change":"2024-09-22T03:20:15.072285+0000","last_active":"2024-09-22T03:21:03.089931+0000","last_peered":"2024-09-22T03:21:03.089931+0000","last_clean":"2024-09-22T03:21:03.089931+0000","last_became_active":"2024-09-22T03:20:15.066796+0000","last_became_peered":"2024-09-22T03:20:15.066796+0000","last_unstale":"2024-09-22T03:21:03.089931+0000","last_undegraded":"2024-09-22T03:21:03.089931+0000","last_fullsized":"2024-09-22T03:21:03.089931+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:19:05.974214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:19:05.974214+0000","last_clean_scrub_stamp":"2024-09-22T03:19:05.974214+0000","objects_scrubbed":0,"log_size":995,"log_dups_size":0,"ondisk_log_size":995,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4325376,"num_objects":66,"num_object_clones":0,"num_object_copies":198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":996,"num_write_kb":7552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":351,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-09-22T03:21:02.403351+0000","last_change":"2024-09-22T03:21:02.404361+0000","last_active":"2024-09-22T03:21:02.403351+0000","last_peered":"2024-09-22T03:21:02.403351+0000","last_clean":"2024-09-22T03:20:18.955898+0000","last_became_active":"2024-09-22T03:21:02.394394+0000","last_became_peered":"2024-09-22T03:21:02.394394+0000","last_unstale":"2024-09-22T03:21:02.403351+0000","last_undegraded":"2024-09-22T03:21:02.403351+0000","last_fullsized":"2024-09-22T03:21:02.403351+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.f","version":"147'925","reported_seq":982,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-09-22T03:21:02.877267+0000","last_change":"2024-09-22T03:21:02.650529+0000","last_active":"2024-09-22T03:21:02.877267+0000","last_peered":"2024-09-22T03:21:02.877267+0000","last_clean":"2024-09-22T03:21:02.877267+0000","last_became_active":"2024-09-22T03:21:02.449362+0000","last_became_peered":"2024-09-22T03:21:02.449362+0000","last_unstale":"2024-09-22T03:21:02.877267+0000","last_undegraded":"2024-09-22T03:21:02.877267+0000","last_fullsized":"2024-09-22T03:21:02.877267+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:19:05.974214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:19:05.974214+0000","last_clean_scrub_stamp":"2024-09-22T03:19:05.974214+0000","objects_scrubbed":0,"log_size":925,"log_dups_size":0,"ondisk_log_size":925,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4931584,"num_objects":76,"num_object_clones":0,"num_object_copies":228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":926,"num_write_kb":7120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":346,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-09-22T03:21:02.426602+0000","last_change":"2024-09-22T03:21:02.427611+0000","last_active":"2024-09-22T03:21:02.426602+0000","last_peered":"2024-09-22T03:21:02.426602+0000","last_clean":"2024-09-22T03:20:18.964527+0000","last_became_active":"2024-09-22T03:21:02.418856+0000","last_became_peered":"2024-09-22T03:21:02.418856+0000","last_unstale":"2024-09-22T03:21:02.426602+0000","last_undegraded":"2024-09-22T03:21:02.426602+0000","last_fullsized":"2024-09-22T03:21:02.426602+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":63914008,"num_objects":977,"num_object_clones":0,"num_object_copies":2931,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":15462,"num_write_kb":116881,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":90112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":94412800,"data_stored":188858440,"data_compressed":9751589,"data_compressed_allocated":94400512,"data_compressed_original":188858368,"omap_allocated":0,"internal_metadata":0},"log_size":15450,"ondisk_log_size":15450,"up":48,"acting":48,"num_store_stats":4},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1180736,"data_compressed":59924,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":146,"seq":627065225220,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":45980,"kb_used_data":11308,"kb_used_omap":0,"kb_used_meta":34624,"kb_avail":94325860,"statfs":{"total":96636764160,"available":96589680640,"internally_reserved":0,"allocated":11579392,"data_stored":21163256,"data_compressed":1066806,"data_compressed_allocated":10330112,"data_compressed_original":20660224,"omap_allocated":0,"internal_metadata":35454976},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":100,"seq":429496729884,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":85400,"kb_used_data":26440,"kb_used_omap":0,"kb_used_meta":58944,"kb_avail":94286440,"statfs":{"total":96636764160,"available":96549314560,"internally_reserved":0,"allocated":27074560,"data_stored":52659744,"data_compressed":2693169,"data_compressed_allocated":25829376,"data_compressed_original":52150272,"omap_allocated":0,"internal_metadata":60358656},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":138,"seq":592705486942,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":72996,"kb_used_data":26964,"kb_used_omap":0,"kb_used_meta":46016,"kb_avail":94298844,"statfs":{"total":96636764160,"available":96562016256,"internally_reserved":0,"allocated":27611136,"data_stored":53728606,"data_compressed":2748160,"data_compressed_allocated":26361856,"data_compressed_original":53215232,"omap_allocated":0,"internal_metadata":47120384},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":142,"seq":609885356081,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":78256,"kb_used_data":35520,"kb_used_omap":0,"kb_used_meta":42688,"kb_avail":94293584,"statfs":{"total":96636764160,"available":96556630016,"internally_reserved":0,"allocated":36372480,"data_stored":71295556,"data_compressed":3654820,"data_compressed_allocated":35115008,"data_compressed_original":70778880,"omap_allocated":0,"internal_metadata":43712512},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29962,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29962,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":33603584,"data_stored":67256344,"data_compressed":3472578,"data_compressed_allocated":33599488,"data_compressed_original":67256320,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":25239552,"data_stored":50479104,"data_compressed":2606526,"data_compressed_allocated":25239552,"data_compressed_original":50479104,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":25792512,"data_stored":51576856,"data_compressed":2663207,"data_compressed_allocated":25788416,"data_compressed_original":51576832,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":9777152,"data_stored":19546136,"data_compressed":1009278,"data_compressed_allocated":9773056,"data_compressed_original":19546112,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-22T03:21:09.158 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-22T03:21:09.273 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:21:09.269+0000 7f419c82c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:21:09.276 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:21:09.274+0000 7f419c82c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:21:10.037 INFO:tasks.ceph.ceph_manager.ceph:removing pool_name unique_pool_6 2024-09-22T03:21:10.037 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm unique_pool_6 unique_pool_6 --yes-i-really-really-mean-it 2024-09-22T03:21:10.149 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:21:10.140+0000 7f94a04d7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:21:10.153 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:21:10.151+0000 7f949f275640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:21:10.970 INFO:teuthology.orchestra.run.smithi153.stderr:pool 'unique_pool_6' removed 2024-09-22T03:21:10.990 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-09-22T03:21:10.990 INFO:tasks.radosbench:Beginning radosbench... 2024-09-22T03:21:10.990 INFO:tasks.ceph.ceph_manager.ceph:creating pool_name unique_pool_7 2024-09-22T03:21:10.990 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create unique_pool_7 16 2024-09-22T03:21:11.137 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:21:11.127+0000 7fbde1717640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:21:11.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:21:11.139+0000 7fbde1717640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:21:11.347 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-22T03:21:11.449 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:21:11.446+0000 7fc532117640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:21:11.453 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:21:11.451+0000 7fc532117640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:21:11.971 INFO:teuthology.orchestra.run.smithi153.stderr:pool 'unique_pool_7' created 2024-09-22T03:21:11.994 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool application enable unique_pool_7 rados --yes-i-really-mean-it || true 2024-09-22T03:21:12.118 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:21:12.108+0000 7fe929917640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:21:12.121 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:21:12.120+0000 7fe929917640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:21:12.494 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [2, 1, 0, 3] 2024-09-22T03:21:12.494 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-22T03:21:12.495 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-22T03:21:12.600 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:21:12.595+0000 7f303b197640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:21:12.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:21:12.601+0000 7f303b197640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:21:12.778 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-22T03:21:12.779 INFO:teuthology.orchestra.run.smithi153.stderr:dumped all 2024-09-22T03:21:12.789 INFO:teuthology.orchestra.run.smithi153.stdout:{"pg_ready":true,"pg_map":{"version":538,"stamp":"2024-09-22T03:21:12.274175+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":139,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":27,"acting":27,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":162428,"kb_used_data":5052,"kb_used_omap":0,"kb_used_meta":157312,"kb_avail":377324932,"statfs":{"total":386547056640,"available":386380730368,"internally_reserved":0,"allocated":5173248,"data_stored":3786378,"data_compressed":89886,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":161087488},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-88162304,"num_objects":-1346,"num_object_clones":0,"num_object_copies":-4038,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-14893,"num_write_kb":-115472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-4,"num_bytes_recovered":-90112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001421"},"pg_stats":[{"pgid":"10.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-22T03:21:11.969997+0000","last_change":"2024-09-22T03:21:11.969997+0000","last_active":"2024-09-22T03:21:11.969997+0000","last_peered":"2024-09-22T03:21:11.969997+0000","last_clean":"2024-09-22T03:21:11.969997+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-22T03:21:11.969997+0000","last_undegraded":"2024-09-22T03:21:11.969997+0000","last_fullsized":"2024-09-22T03:21:11.969997+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:21:11.969997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:21:11.969997+0000","last_clean_scrub_stamp":"2024-09-22T03:21:11.969997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-22T03:21:11.969997+0000","last_change":"2024-09-22T03:21:11.969997+0000","last_active":"2024-09-22T03:21:11.969997+0000","last_peered":"2024-09-22T03:21:11.969997+0000","last_clean":"2024-09-22T03:21:11.969997+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-22T03:21:11.969997+0000","last_undegraded":"2024-09-22T03:21:11.969997+0000","last_fullsized":"2024-09-22T03:21:11.969997+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:21:11.969997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:21:11.969997+0000","last_clean_scrub_stamp":"2024-09-22T03:21:11.969997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-22T03:21:11.969997+0000","last_change":"2024-09-22T03:21:11.969997+0000","last_active":"2024-09-22T03:21:11.969997+0000","last_peered":"2024-09-22T03:21:11.969997+0000","last_clean":"2024-09-22T03:21:11.969997+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-22T03:21:11.969997+0000","last_undegraded":"2024-09-22T03:21:11.969997+0000","last_fullsized":"2024-09-22T03:21:11.969997+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:21:11.969997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:21:11.969997+0000","last_clean_scrub_stamp":"2024-09-22T03:21:11.969997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-22T03:21:11.969997+0000","last_change":"2024-09-22T03:21:11.969997+0000","last_active":"2024-09-22T03:21:11.969997+0000","last_peered":"2024-09-22T03:21:11.969997+0000","last_clean":"2024-09-22T03:21:11.969997+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-22T03:21:11.969997+0000","last_undegraded":"2024-09-22T03:21:11.969997+0000","last_fullsized":"2024-09-22T03:21:11.969997+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:21:11.969997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:21:11.969997+0000","last_clean_scrub_stamp":"2024-09-22T03:21:11.969997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-22T03:21:11.969997+0000","last_change":"2024-09-22T03:21:11.969997+0000","last_active":"2024-09-22T03:21:11.969997+0000","last_peered":"2024-09-22T03:21:11.969997+0000","last_clean":"2024-09-22T03:21:11.969997+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-22T03:21:11.969997+0000","last_undegraded":"2024-09-22T03:21:11.969997+0000","last_fullsized":"2024-09-22T03:21:11.969997+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:21:11.969997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:21:11.969997+0000","last_clean_scrub_stamp":"2024-09-22T03:21:11.969997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-22T03:21:11.969997+0000","last_change":"2024-09-22T03:21:11.969997+0000","last_active":"2024-09-22T03:21:11.969997+0000","last_peered":"2024-09-22T03:21:11.969997+0000","last_clean":"2024-09-22T03:21:11.969997+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-22T03:21:11.969997+0000","last_undegraded":"2024-09-22T03:21:11.969997+0000","last_fullsized":"2024-09-22T03:21:11.969997+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:21:11.969997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:21:11.969997+0000","last_clean_scrub_stamp":"2024-09-22T03:21:11.969997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-22T03:21:11.969997+0000","last_change":"2024-09-22T03:21:11.969997+0000","last_active":"2024-09-22T03:21:11.969997+0000","last_peered":"2024-09-22T03:21:11.969997+0000","last_clean":"2024-09-22T03:21:11.969997+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-22T03:21:11.969997+0000","last_undegraded":"2024-09-22T03:21:11.969997+0000","last_fullsized":"2024-09-22T03:21:11.969997+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:21:11.969997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:21:11.969997+0000","last_clean_scrub_stamp":"2024-09-22T03:21:11.969997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-22T03:21:11.969997+0000","last_change":"2024-09-22T03:21:11.969997+0000","last_active":"2024-09-22T03:21:11.969997+0000","last_peered":"2024-09-22T03:21:11.969997+0000","last_clean":"2024-09-22T03:21:11.969997+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-22T03:21:11.969997+0000","last_undegraded":"2024-09-22T03:21:11.969997+0000","last_fullsized":"2024-09-22T03:21:11.969997+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:21:11.969997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:21:11.969997+0000","last_clean_scrub_stamp":"2024-09-22T03:21:11.969997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-22T03:21:11.969997+0000","last_change":"2024-09-22T03:21:11.969997+0000","last_active":"2024-09-22T03:21:11.969997+0000","last_peered":"2024-09-22T03:21:11.969997+0000","last_clean":"2024-09-22T03:21:11.969997+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-22T03:21:11.969997+0000","last_undegraded":"2024-09-22T03:21:11.969997+0000","last_fullsized":"2024-09-22T03:21:11.969997+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:21:11.969997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:21:11.969997+0000","last_clean_scrub_stamp":"2024-09-22T03:21:11.969997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":211,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-22T03:21:03.982405+0000","last_change":"2024-09-22T03:21:03.985334+0000","last_active":"2024-09-22T03:21:03.982405+0000","last_peered":"2024-09-22T03:21:03.982405+0000","last_clean":"2024-09-22T03:21:03.982405+0000","last_became_active":"2024-09-22T03:21:02.491715+0000","last_became_peered":"2024-09-22T03:21:02.491715+0000","last_unstale":"2024-09-22T03:21:03.982405+0000","last_undegraded":"2024-09-22T03:21:03.982405+0000","last_fullsized":"2024-09-22T03:21:03.982405+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-22T03:21:11.969997+0000","last_change":"2024-09-22T03:21:11.969997+0000","last_active":"2024-09-22T03:21:11.969997+0000","last_peered":"2024-09-22T03:21:11.969997+0000","last_clean":"2024-09-22T03:21:11.969997+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-22T03:21:11.969997+0000","last_undegraded":"2024-09-22T03:21:11.969997+0000","last_fullsized":"2024-09-22T03:21:11.969997+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:21:11.969997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:21:11.969997+0000","last_clean_scrub_stamp":"2024-09-22T03:21:11.969997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":367,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-22T03:21:04.001748+0000","last_change":"2024-09-22T03:20:18.964214+0000","last_active":"2024-09-22T03:21:04.001748+0000","last_peered":"2024-09-22T03:21:04.001748+0000","last_clean":"2024-09-22T03:21:04.001748+0000","last_became_active":"2024-09-22T03:20:15.069435+0000","last_became_peered":"2024-09-22T03:20:15.069435+0000","last_unstale":"2024-09-22T03:21:04.001748+0000","last_undegraded":"2024-09-22T03:21:04.001748+0000","last_fullsized":"2024-09-22T03:21:04.001748+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-22T03:21:11.969997+0000","last_change":"2024-09-22T03:21:11.969997+0000","last_active":"2024-09-22T03:21:11.969997+0000","last_peered":"2024-09-22T03:21:11.969997+0000","last_clean":"2024-09-22T03:21:11.969997+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-22T03:21:11.969997+0000","last_undegraded":"2024-09-22T03:21:11.969997+0000","last_fullsized":"2024-09-22T03:21:11.969997+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:21:11.969997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:21:11.969997+0000","last_clean_scrub_stamp":"2024-09-22T03:21:11.969997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":374,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-22T03:21:10.993561+0000","last_change":"2024-09-22T03:21:03.970220+0000","last_active":"2024-09-22T03:21:10.993561+0000","last_peered":"2024-09-22T03:21:10.993561+0000","last_clean":"2024-09-22T03:21:10.993561+0000","last_became_active":"2024-09-22T03:21:02.396644+0000","last_became_peered":"2024-09-22T03:21:02.396644+0000","last_unstale":"2024-09-22T03:21:10.993561+0000","last_undegraded":"2024-09-22T03:21:10.993561+0000","last_fullsized":"2024-09-22T03:21:10.993561+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-22T03:21:11.969997+0000","last_change":"2024-09-22T03:21:11.969997+0000","last_active":"2024-09-22T03:21:11.969997+0000","last_peered":"2024-09-22T03:21:11.969997+0000","last_clean":"2024-09-22T03:21:11.969997+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-22T03:21:11.969997+0000","last_undegraded":"2024-09-22T03:21:11.969997+0000","last_fullsized":"2024-09-22T03:21:11.969997+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:21:11.969997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:21:11.969997+0000","last_clean_scrub_stamp":"2024-09-22T03:21:11.969997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":241,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-22T03:21:03.983292+0000","last_change":"2024-09-22T03:21:03.989872+0000","last_active":"2024-09-22T03:21:03.983292+0000","last_peered":"2024-09-22T03:21:03.983292+0000","last_clean":"2024-09-22T03:21:03.983292+0000","last_became_active":"2024-09-22T03:21:02.455063+0000","last_became_peered":"2024-09-22T03:21:02.455063+0000","last_unstale":"2024-09-22T03:21:03.983292+0000","last_undegraded":"2024-09-22T03:21:03.983292+0000","last_fullsized":"2024-09-22T03:21:03.983292+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"12'97","reported_seq":487,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-22T03:21:04.002729+0000","last_change":"2024-09-22T03:20:15.058640+0000","last_active":"2024-09-22T03:21:04.002729+0000","last_peered":"2024-09-22T03:21:04.002729+0000","last_clean":"2024-09-22T03:21:04.002729+0000","last_became_active":"2024-09-22T03:20:15.055409+0000","last_became_peered":"2024-09-22T03:20:15.055409+0000","last_unstale":"2024-09-22T03:21:04.002729+0000","last_undegraded":"2024-09-22T03:21:04.002729+0000","last_fullsized":"2024-09-22T03:21:04.002729+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-22T03:21:11.969997+0000","last_change":"2024-09-22T03:21:11.969997+0000","last_active":"2024-09-22T03:21:11.969997+0000","last_peered":"2024-09-22T03:21:11.969997+0000","last_clean":"2024-09-22T03:21:11.969997+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-22T03:21:11.969997+0000","last_undegraded":"2024-09-22T03:21:11.969997+0000","last_fullsized":"2024-09-22T03:21:11.969997+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:21:11.969997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:21:11.969997+0000","last_clean_scrub_stamp":"2024-09-22T03:21:11.969997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":278,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-22T03:21:03.971451+0000","last_change":"2024-09-22T03:20:18.957911+0000","last_active":"2024-09-22T03:21:03.971451+0000","last_peered":"2024-09-22T03:21:03.971451+0000","last_clean":"2024-09-22T03:21:03.971451+0000","last_became_active":"2024-09-22T03:20:15.017488+0000","last_became_peered":"2024-09-22T03:20:15.017488+0000","last_unstale":"2024-09-22T03:21:03.971451+0000","last_undegraded":"2024-09-22T03:21:03.971451+0000","last_fullsized":"2024-09-22T03:21:03.971451+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-22T03:21:11.969997+0000","last_change":"2024-09-22T03:21:11.969997+0000","last_active":"2024-09-22T03:21:11.969997+0000","last_peered":"2024-09-22T03:21:11.969997+0000","last_clean":"2024-09-22T03:21:11.969997+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-22T03:21:11.969997+0000","last_undegraded":"2024-09-22T03:21:11.969997+0000","last_fullsized":"2024-09-22T03:21:11.969997+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:21:11.969997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:21:11.969997+0000","last_clean_scrub_stamp":"2024-09-22T03:21:11.969997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":349,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-22T03:21:10.998832+0000","last_change":"2024-09-22T03:21:03.970088+0000","last_active":"2024-09-22T03:21:10.998832+0000","last_peered":"2024-09-22T03:21:10.998832+0000","last_clean":"2024-09-22T03:21:10.998832+0000","last_became_active":"2024-09-22T03:21:02.418956+0000","last_became_peered":"2024-09-22T03:21:02.418956+0000","last_unstale":"2024-09-22T03:21:10.998832+0000","last_undegraded":"2024-09-22T03:21:10.998832+0000","last_fullsized":"2024-09-22T03:21:10.998832+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-22T03:21:11.969997+0000","last_change":"2024-09-22T03:21:11.969997+0000","last_active":"2024-09-22T03:21:11.969997+0000","last_peered":"2024-09-22T03:21:11.969997+0000","last_clean":"2024-09-22T03:21:11.969997+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-22T03:21:11.969997+0000","last_undegraded":"2024-09-22T03:21:11.969997+0000","last_fullsized":"2024-09-22T03:21:11.969997+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:21:11.969997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:21:11.969997+0000","last_clean_scrub_stamp":"2024-09-22T03:21:11.969997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":355,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-22T03:21:03.970980+0000","last_change":"2024-09-22T03:21:03.975030+0000","last_active":"2024-09-22T03:21:03.970980+0000","last_peered":"2024-09-22T03:21:03.970980+0000","last_clean":"2024-09-22T03:21:03.970980+0000","last_became_active":"2024-09-22T03:21:02.394394+0000","last_became_peered":"2024-09-22T03:21:02.394394+0000","last_unstale":"2024-09-22T03:21:03.970980+0000","last_undegraded":"2024-09-22T03:21:03.970980+0000","last_fullsized":"2024-09-22T03:21:03.970980+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-22T03:21:11.969997+0000","last_change":"2024-09-22T03:21:11.969997+0000","last_active":"2024-09-22T03:21:11.969997+0000","last_peered":"2024-09-22T03:21:11.969997+0000","last_clean":"2024-09-22T03:21:11.969997+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-22T03:21:11.969997+0000","last_undegraded":"2024-09-22T03:21:11.969997+0000","last_fullsized":"2024-09-22T03:21:11.969997+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:21:11.969997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:21:11.969997+0000","last_clean_scrub_stamp":"2024-09-22T03:21:11.969997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":350,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-22T03:21:03.975299+0000","last_change":"2024-09-22T03:21:03.989877+0000","last_active":"2024-09-22T03:21:03.975299+0000","last_peered":"2024-09-22T03:21:03.975299+0000","last_clean":"2024-09-22T03:21:03.975299+0000","last_became_active":"2024-09-22T03:21:02.418856+0000","last_became_peered":"2024-09-22T03:21:02.418856+0000","last_unstale":"2024-09-22T03:21:03.975299+0000","last_undegraded":"2024-09-22T03:21:03.975299+0000","last_fullsized":"2024-09-22T03:21:03.975299+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":16,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1180736,"data_compressed":59924,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":146,"seq":627065225225,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":35856,"kb_used_data":1216,"kb_used_omap":0,"kb_used_meta":34624,"kb_avail":94335984,"statfs":{"total":96636764160,"available":96600047616,"internally_reserved":0,"allocated":1245184,"data_stored":503008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":35454976},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":100,"seq":429496729894,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":64408,"kb_used_data":1288,"kb_used_omap":0,"kb_used_meta":63104,"kb_avail":94307432,"statfs":{"total":96636764160,"available":96570810368,"internally_reserved":0,"allocated":1318912,"data_stored":1096637,"data_compressed":29962,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64618496},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":138,"seq":592705486947,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":30668,"kb_used_data":1276,"kb_used_omap":0,"kb_used_meta":29376,"kb_avail":94341172,"statfs":{"total":96636764160,"available":96605360128,"internally_reserved":0,"allocated":1306624,"data_stored":1093376,"data_compressed":29962,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":30081024},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":142,"seq":609885356086,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31496,"kb_used_data":1272,"kb_used_omap":0,"kb_used_meta":30208,"kb_avail":94340344,"statfs":{"total":96636764160,"available":96604512256,"internally_reserved":0,"allocated":1302528,"data_stored":1093357,"data_compressed":29962,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":30932992},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29962,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29962,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-22T03:21:12.982 INFO:teuthology.orchestra.run.smithi153.stderr:enabled application 'rados' on pool 'unique_pool_7' 2024-09-22T03:21:13.518 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-22T03:21:13.622 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:21:13.619+0000 7f2cd6f03640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:21:13.625 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:21:13.623+0000 7f2cd6f03640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:21:14.008 DEBUG:teuthology.orchestra.run.smithi153:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 8192 --object-size 65536 -p unique_pool_7 bench 90 write' 2024-09-22T03:21:14.010 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-09-22T03:21:14.010 INFO:teuthology.orchestra.run:waiting for 3000 2024-09-22T03:21:14.065 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:hints = 1 2024-09-22T03:21:14.066 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:Maintaining 16 concurrent writes of 8192 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-09-22T03:21:14.066 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:Object prefix: benchmark_data_smithi153_46957 2024-09-22T03:21:14.066 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-22T03:21:14.066 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 0 0 0 0 0 0 - 0 2024-09-22T03:21:15.065 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 1 16 370 354 2.76546 2.76562 0.0268391 0.0432186 2024-09-22T03:21:15.691 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-22T03:21:15.831 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:21:15.828+0000 7fee9e24f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:21:15.835 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:21:15.833+0000 7fee9e24f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:21:16.066 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 2 16 734 718 2.8045 2.84375 0.0367916 0.0436607 2024-09-22T03:21:17.066 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 3 16 1115 1099 2.86177 2.97656 0.0182144 0.0433324 2024-09-22T03:21:17.791 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [2, 1, 0, 3] 2024-09-22T03:21:17.791 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-22T03:21:17.791 INFO:tasks.thrashosds.thrasher:Killing osd 3, live_osds are [2, 1, 0, 3] 2024-09-22T03:21:17.792 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-09-22T03:21:17.792 INFO:teuthology.orchestra.run:waiting for 300 2024-09-22T03:21:17.982 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-22T03:21:18.066 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 4 16 1404 1388 2.71074 2.25781 0.0239373 0.042968 2024-09-22T03:21:18.084 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:21:18.081+0000 7fba45858640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:21:18.087 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:21:18.086+0000 7fba45858640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:21:19.066 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 5 16 1404 1388 2.16856 0 - 0.042968 2024-09-22T03:21:20.023 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~0s 2024-09-22T03:21:20.066 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 6 16 1404 1388 1.80712 0 - 0.042968 2024-09-22T03:21:21.066 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 7 16 1404 1388 1.54896 0 - 0.042968 2024-09-22T03:21:22.066 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 8 16 1404 1388 1.35533 0 - 0.042968 2024-09-22T03:21:23.066 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 9 16 1404 1388 1.20473 0 - 0.042968 2024-09-22T03:21:23.893 INFO:tasks.ceph.osd.3:Stopped 2024-09-22T03:21:24.067 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 10 16 1404 1388 1.08426 0 - 0.042968 2024-09-22T03:21:25.067 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 11 16 1404 1388 0.985687 0 - 0.042968 2024-09-22T03:21:25.528 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 has been restored 2024-09-22T03:21:26.067 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 12 16 1404 1388 0.903544 0 - 0.042968 2024-09-22T03:21:27.067 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 13 16 1404 1388 0.83404 0 - 0.042968 2024-09-22T03:21:28.067 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 14 16 1404 1388 0.774465 0 - 0.042968 2024-09-22T03:21:28.894 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [3] live_osds: [2, 1, 0] 2024-09-22T03:21:28.894 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-22T03:21:28.894 INFO:tasks.thrashosds.thrasher:Killing osd 2, live_osds are [2, 1, 0] 2024-09-22T03:21:28.894 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-09-22T03:21:28.894 INFO:teuthology.orchestra.run:waiting for 300 2024-09-22T03:21:29.067 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 15 16 1404 1388 0.722834 0 - 0.042968 2024-09-22T03:21:30.067 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 16 16 1404 1388 0.677656 0 - 0.042968 2024-09-22T03:21:30.931 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~0s 2024-09-22T03:21:31.067 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 17 16 1404 1388 0.637792 0 - 0.042968 2024-09-22T03:21:32.068 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 18 16 1404 1388 0.602359 0 - 0.042968 2024-09-22T03:21:33.068 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 19 16 1404 1388 0.570655 0 - 0.042968 2024-09-22T03:21:34.068 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:2024-09-22T03:21:34.067402+0000 min lat: 0.00896526 max lat: 0.121578 avg lat: 0.042968 2024-09-22T03:21:34.068 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-22T03:21:34.068 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 20 16 1404 1388 0.542123 0 - 0.042968 2024-09-22T03:21:34.995 INFO:tasks.ceph.osd.2:Stopped 2024-09-22T03:21:35.068 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 21 16 1404 1388 0.516307 0 - 0.042968 2024-09-22T03:21:36.068 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 22 16 1404 1388 0.492838 0 - 0.042968 2024-09-22T03:21:36.334 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 has been restored 2024-09-22T03:21:37.068 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 23 16 1404 1388 0.47141 0 - 0.042968 2024-09-22T03:21:38.068 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 24 16 1404 1388 0.451768 0 - 0.042968 2024-09-22T03:21:39.068 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 25 16 1404 1388 0.433698 0 - 0.042968 2024-09-22T03:21:39.996 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [3, 2] live_osds: [1, 0] 2024-09-22T03:21:39.997 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-22T03:21:39.997 INFO:tasks.thrashosds.thrasher:Reviving osd 3 2024-09-22T03:21:39.997 INFO:tasks.ceph.osd.3:Restarting daemon 2024-09-22T03:21:39.997 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2>> /var/log/ceph/ceph-osd.3.log 2024-09-22T03:21:40.000 INFO:tasks.ceph.osd.3:Started 2024-09-22T03:21:40.000 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-22T03:21:40.069 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 26 16 1404 1388 0.417016 0 - 0.042968 2024-09-22T03:21:40.110 INFO:teuthology.orchestra.run.smithi196.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-09-22T03:21:40.113 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-22T03:21:40.114 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-09-22T03:21:41.069 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 27 16 1404 1388 0.401571 0 - 0.042968 2024-09-22T03:21:42.069 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 28 16 1404 1388 0.387229 0 - 0.042968 2024-09-22T03:21:43.069 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 29 16 1404 1388 0.373876 0 - 0.042968 2024-09-22T03:21:44.069 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 30 16 1404 1388 0.361414 0 - 0.042968 2024-09-22T03:21:44.986 INFO:teuthology.orchestra.run.smithi153.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-09-22T03:21:44.991 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-22T03:21:44.991 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-22T03:21:45.069 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 31 16 1404 1388 0.349755 0 - 0.042968 2024-09-22T03:21:45.115 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-22T03:21:45.210 INFO:teuthology.orchestra.run.smithi196.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-09-22T03:21:45.213 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-22T03:21:45.213 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-09-22T03:21:46.069 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 32 16 1404 1388 0.338826 0 - 0.042968 2024-09-22T03:21:46.992 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-22T03:21:47.069 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 33 16 1404 1388 0.328559 0 - 0.042968 2024-09-22T03:21:47.106 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:21:47.102+0000 7f8f2d62c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:21:47.109 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:21:47.107+0000 7f8f2d62c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:21:48.069 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 34 16 1404 1388 0.318895 0 - 0.042968 2024-09-22T03:21:49.070 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 35 16 1404 1388 0.309784 0 - 0.042968 2024-09-22T03:21:50.070 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 36 16 1404 1388 0.301179 0 - 0.042968 2024-09-22T03:21:50.214 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-22T03:21:50.325 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-09-22T03:21:50.325 INFO:teuthology.orchestra.run.smithi196.stdout: "ops_in_flight": [] 2024-09-22T03:21:50.325 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-09-22T03:21:51.070 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 37 16 1404 1388 0.293039 0 - 0.042968 2024-09-22T03:21:52.070 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 38 16 1404 1388 0.285327 0 - 0.042968 2024-09-22T03:21:53.070 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 39 16 1404 1388 0.278011 0 - 0.042968 2024-09-22T03:21:54.070 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:2024-09-22T03:21:54.069961+0000 min lat: 0.00896526 max lat: 0.121578 avg lat: 0.042968 2024-09-22T03:21:54.070 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-22T03:21:54.071 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 40 16 1404 1388 0.27106 0 - 0.042968 2024-09-22T03:21:55.071 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 41 16 1404 1388 0.264449 0 - 0.042968 2024-09-22T03:21:55.335 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [2] live_osds: [1, 0, 3] 2024-09-22T03:21:55.335 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-22T03:21:55.335 INFO:tasks.thrashosds.thrasher:Killing osd 0, live_osds are [1, 0, 3] 2024-09-22T03:21:55.336 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-09-22T03:21:55.336 INFO:teuthology.orchestra.run:waiting for 300 2024-09-22T03:21:56.071 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 42 16 1404 1388 0.258153 0 - 0.042968 2024-09-22T03:21:56.973 INFO:teuthology.orchestra.run.smithi153.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-09-22T03:21:57.028 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-22T03:21:57.029 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-22T03:21:57.071 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 43 16 1404 1388 0.252149 0 - 0.042968 2024-09-22T03:21:58.071 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 44 16 1416 1400 0.248549 0.00234375 0.0240525 0.272671 2024-09-22T03:21:58.251 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~0s 2024-09-22T03:21:59.030 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-22T03:21:59.071 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 45 16 1416 1400 0.243026 0 - 0.272671 2024-09-22T03:21:59.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:21:59.142+0000 7f9235f17640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:21:59.149 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:21:59.147+0000 7f9235f17640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:22:00.071 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 46 16 1416 1400 0.237743 0 - 0.272671 2024-09-22T03:22:01.071 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 47 16 1416 1400 0.232684 0 - 0.272671 2024-09-22T03:22:01.437 INFO:tasks.ceph.osd.0:Stopped 2024-09-22T03:22:02.071 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 48 16 1416 1400 0.227836 0 - 0.272671 2024-09-22T03:22:03.071 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 49 16 1416 1400 0.223187 0 - 0.272671 2024-09-22T03:22:03.655 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 has been restored 2024-09-22T03:22:04.072 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 50 16 1416 1400 0.218723 0 - 0.272671 2024-09-22T03:22:05.072 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 51 16 1416 1400 0.214434 0 - 0.272671 2024-09-22T03:22:06.072 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 52 16 1416 1400 0.21031 0 - 0.272671 2024-09-22T03:22:06.438 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [2, 0] live_osds: [1, 3] 2024-09-22T03:22:06.438 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-22T03:22:06.438 INFO:tasks.thrashosds.thrasher:Reviving osd 2 2024-09-22T03:22:06.438 INFO:tasks.ceph.osd.2:Restarting daemon 2024-09-22T03:22:06.438 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2>> /var/log/ceph/ceph-osd.2.log 2024-09-22T03:22:06.441 INFO:tasks.ceph.osd.2:Started 2024-09-22T03:22:06.441 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-22T03:22:06.551 INFO:teuthology.orchestra.run.smithi196.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-09-22T03:22:06.554 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-22T03:22:06.554 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2024-09-22T03:22:07.072 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 53 16 1416 1400 0.206342 0 - 0.272671 2024-09-22T03:22:08.072 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 54 16 1416 1400 0.202521 0 - 0.272671 2024-09-22T03:22:09.072 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 55 16 1416 1400 0.198838 0 - 0.272671 2024-09-22T03:22:10.072 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 56 16 1416 1400 0.195288 0 - 0.272671 2024-09-22T03:22:11.073 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 57 16 1416 1400 0.191862 0 - 0.272671 2024-09-22T03:22:11.557 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-22T03:22:11.653 INFO:teuthology.orchestra.run.smithi196.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-09-22T03:22:11.655 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-22T03:22:11.656 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2024-09-22T03:22:12.073 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 58 16 1416 1400 0.188554 0 - 0.272671 2024-09-22T03:22:13.073 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 59 16 1416 1400 0.185358 0 - 0.272671 2024-09-22T03:22:14.073 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:2024-09-22T03:22:14.072698+0000 min lat: 0.00896526 max lat: 40.2815 avg lat: 0.272671 2024-09-22T03:22:14.073 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-22T03:22:14.073 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 60 16 1416 1400 0.182268 0 - 0.272671 2024-09-22T03:22:15.073 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 61 16 1416 1400 0.17928 0 - 0.272671 2024-09-22T03:22:16.073 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 62 16 1416 1400 0.176389 0 - 0.272671 2024-09-22T03:22:16.657 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-22T03:22:16.759 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-09-22T03:22:16.759 INFO:teuthology.orchestra.run.smithi196.stdout: "ops_in_flight": [ 2024-09-22T03:22:16.759 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-22T03:22:16.759 INFO:teuthology.orchestra.run.smithi196.stdout: "type": "client_request", 2024-09-22T03:22:16.760 INFO:teuthology.orchestra.run.smithi196.stdout: "id": 15, 2024-09-22T03:22:16.760 INFO:teuthology.orchestra.run.smithi196.stdout: "detail": { 2024-09-22T03:22:16.760 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:22:16.760 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:22:16.760 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:22:15.347334+0000" 2024-09-22T03:22:16.760 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.760 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:22:16.760 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:22:16.760 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:22:15.347362+0000" 2024-09-22T03:22:16.760 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.760 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:22:16.760 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:16.760 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.760 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.761 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:22:16.761 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:22:16.761 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:22:15.360845+0000" 2024-09-22T03:22:16.761 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.761 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:22:16.761 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:16.761 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.761 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.761 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:22:16.761 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:22:16.761 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:22:15.361196+0000" 2024-09-22T03:22:16.761 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.761 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:22:16.762 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:16.762 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.762 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.762 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:22:16.762 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:22:16.762 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:22:15.458725+0000" 2024-09-22T03:22:16.762 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.762 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:22:16.762 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:16.762 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.762 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.762 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:22:16.762 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:22:16.763 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:22:15.459743+0000" 2024-09-22T03:22:16.763 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.763 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:22:16.763 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:16.763 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.763 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.763 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:22:16.763 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:22:16.763 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.763 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.763 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:22:16.763 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:16.764 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.764 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.764 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:22:16.764 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:22:16.764 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.764 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.764 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:22:16.764 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:22:16.764 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:22:15.460022+0000" 2024-09-22T03:22:16.764 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.764 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:22:16.764 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:16.764 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.765 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.765 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:22:16.765 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:22:16.765 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.765 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.765 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:22:16.765 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:16.765 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.765 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.765 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:22:16.765 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:22:16.765 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:22:15.460399+0000" 2024-09-22T03:22:16.765 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.766 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:22:16.766 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:16.766 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.766 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.766 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:22:16.766 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:22:16.766 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:22:15.460505+0000", 2024-09-22T03:22:16.766 INFO:teuthology.orchestra.run.smithi196.stdout: "blocker": { 2024-09-22T03:22:16.766 INFO:teuthology.orchestra.run.smithi196.stdout: "op_type": "PGActivationBlocker", 2024-09-22T03:22:16.766 INFO:teuthology.orchestra.run.smithi196.stdout: "detail": { 2024-09-22T03:22:16.766 INFO:teuthology.orchestra.run.smithi196.stdout: "pgid": "10.8" 2024-09-22T03:22:16.766 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:22:16.766 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:22:16.766 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.767 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:22:16.767 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:16.767 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.767 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.767 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:22:16.767 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:22:16.767 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.767 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.767 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:22:16.767 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:16.767 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.767 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.767 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:22:16.768 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:22:16.768 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.768 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.768 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:22:16.768 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:16.768 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.768 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.768 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:22:16.768 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:22:16.768 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.768 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.768 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:22:16.768 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:16.768 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.769 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.769 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:22:16.769 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:22:16.769 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.769 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.769 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:22:16.769 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:16.769 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.769 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.769 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:22:16.769 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:22:16.769 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.769 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.769 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:22:16.769 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:16.770 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.770 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.770 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:22:16.770 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:22:16.770 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.770 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.770 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:22:16.770 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:16.770 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.770 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.770 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:22:16.770 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:22:16.771 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.771 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.771 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:22:16.771 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:16.771 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.771 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.771 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:22:16.771 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:22:16.771 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.771 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.771 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:22:16.771 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:16.771 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.772 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.772 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:22:16.772 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:22:16.772 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.772 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.772 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:22:16.772 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:16.772 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.772 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.772 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:22:16.772 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:22:16.772 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.772 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:22:16.772 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:22:16.773 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.773 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-22T03:22:16.773 INFO:teuthology.orchestra.run.smithi196.stdout: "type": "client_request", 2024-09-22T03:22:16.773 INFO:teuthology.orchestra.run.smithi196.stdout: "id": 16, 2024-09-22T03:22:16.773 INFO:teuthology.orchestra.run.smithi196.stdout: "detail": { 2024-09-22T03:22:16.773 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:22:16.773 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:22:16.773 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:22:15.347649+0000" 2024-09-22T03:22:16.773 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.773 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:22:16.773 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:22:16.773 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:22:15.347662+0000" 2024-09-22T03:22:16.773 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.774 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:22:16.774 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:16.774 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.774 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.774 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:22:16.774 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:22:16.774 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:22:15.361369+0000" 2024-09-22T03:22:16.774 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.774 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:22:16.774 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:16.774 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.774 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.774 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:22:16.775 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:22:16.775 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.775 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.775 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:22:16.775 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:16.775 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.775 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.775 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:22:16.775 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:22:16.775 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:22:15.459044+0000" 2024-09-22T03:22:16.775 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.775 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:22:16.775 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:16.776 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.776 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.776 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:22:16.776 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:22:16.776 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:22:15.459784+0000" 2024-09-22T03:22:16.776 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.776 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:22:16.776 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:16.776 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.776 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.776 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:22:16.776 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:22:16.777 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.777 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.777 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:22:16.777 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:16.777 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.777 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.777 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:22:16.777 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:22:16.777 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.777 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.777 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:22:16.777 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:22:16.777 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:22:15.460256+0000" 2024-09-22T03:22:16.778 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.778 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:22:16.778 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:16.778 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.778 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.778 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:22:16.778 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:22:16.778 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.778 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.778 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:22:16.778 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:16.778 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.778 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.778 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:22:16.779 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:22:16.779 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:22:15.460637+0000", 2024-09-22T03:22:16.779 INFO:teuthology.orchestra.run.smithi196.stdout: "blocker": { 2024-09-22T03:22:16.779 INFO:teuthology.orchestra.run.smithi196.stdout: "op_type": "CommonPGPipeline:::wait_for_active", 2024-09-22T03:22:16.779 INFO:teuthology.orchestra.run.smithi196.stdout: "detail": { 2024-09-22T03:22:16.779 INFO:teuthology.orchestra.run.smithi196.stdout: "waiting": 1, 2024-09-22T03:22:16.779 INFO:teuthology.orchestra.run.smithi196.stdout: "held_by_operation_id": 15 2024-09-22T03:22:16.779 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:22:16.779 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:22:16.779 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.779 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:22:16.779 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:16.779 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.780 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.780 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:22:16.780 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:22:16.780 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.780 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.780 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:22:16.780 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:16.780 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.780 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.780 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:22:16.780 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:22:16.780 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.780 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.780 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:22:16.781 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:16.781 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.781 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.781 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:22:16.781 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:22:16.781 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.781 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.781 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:22:16.781 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:16.781 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.781 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.781 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:22:16.781 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:22:16.781 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.782 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.782 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:22:16.782 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:16.782 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.782 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.782 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:22:16.782 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:22:16.782 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.782 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.782 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:22:16.782 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:16.782 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.782 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.782 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:22:16.783 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:22:16.783 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.783 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.783 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:22:16.783 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:16.783 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.783 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.783 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:22:16.783 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:22:16.783 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.783 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.783 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:22:16.783 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:16.784 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.784 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.784 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:22:16.784 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:22:16.784 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.784 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.784 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:22:16.784 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:16.784 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.784 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.784 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:22:16.784 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:22:16.785 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.785 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.785 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:22:16.785 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:16.785 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.785 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.785 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:22:16.785 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:22:16.785 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.785 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.785 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:22:16.785 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:16.785 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.785 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.786 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:22:16.786 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:22:16.786 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.786 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:22:16.786 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:22:16.786 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.786 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-22T03:22:16.786 INFO:teuthology.orchestra.run.smithi196.stdout: "type": "client_request", 2024-09-22T03:22:16.786 INFO:teuthology.orchestra.run.smithi196.stdout: "id": 17, 2024-09-22T03:22:16.786 INFO:teuthology.orchestra.run.smithi196.stdout: "detail": { 2024-09-22T03:22:16.786 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:22:16.786 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:22:16.786 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:22:15.347883+0000" 2024-09-22T03:22:16.787 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.787 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:22:16.787 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:22:16.787 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:22:15.347888+0000" 2024-09-22T03:22:16.787 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.787 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:22:16.787 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:16.787 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.787 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.787 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:22:16.787 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:22:16.787 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:22:15.459120+0000" 2024-09-22T03:22:16.787 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.787 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:22:16.787 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:16.788 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.788 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.788 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:22:16.788 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:22:16.788 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.788 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.788 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:22:16.788 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:16.788 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.788 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.788 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:22:16.788 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:22:16.788 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:22:15.459621+0000" 2024-09-22T03:22:16.789 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.789 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:22:16.789 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:16.789 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.789 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.789 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:22:16.789 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:22:16.789 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:22:15.460725+0000" 2024-09-22T03:22:16.789 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.789 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:22:16.789 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:16.789 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.789 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.789 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:22:16.790 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:22:16.790 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.790 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.790 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:22:16.790 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:16.790 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.790 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.790 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:22:16.790 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:22:16.790 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.790 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.790 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:22:16.791 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:22:16.791 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:22:15.460940+0000", 2024-09-22T03:22:16.791 INFO:teuthology.orchestra.run.smithi196.stdout: "blocker": { 2024-09-22T03:22:16.791 INFO:teuthology.orchestra.run.smithi196.stdout: "op_type": "ClientRequest::PGPipeline::await_map", 2024-09-22T03:22:16.791 INFO:teuthology.orchestra.run.smithi196.stdout: "detail": { 2024-09-22T03:22:16.791 INFO:teuthology.orchestra.run.smithi196.stdout: "waiting": 1, 2024-09-22T03:22:16.791 INFO:teuthology.orchestra.run.smithi196.stdout: "held_by_operation_id": 16 2024-09-22T03:22:16.791 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:22:16.791 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:22:16.791 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.791 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:22:16.791 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:16.791 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.791 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.792 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:22:16.792 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:22:16.792 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.792 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.792 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:22:16.792 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:16.792 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.792 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.792 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:22:16.792 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:22:16.792 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.792 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.792 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:22:16.792 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:16.792 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.793 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.793 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:22:16.793 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:22:16.793 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.793 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.793 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:22:16.793 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:16.793 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.793 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.793 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:22:16.793 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:22:16.793 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.793 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.793 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:22:16.794 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:16.794 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.794 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.794 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:22:16.794 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:22:16.794 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.794 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.794 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:22:16.794 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:16.794 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.794 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.794 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:22:16.794 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:22:16.795 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.795 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.795 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:22:16.795 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:16.795 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.795 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.795 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:22:16.795 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:22:16.795 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.795 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.795 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:22:16.795 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:16.795 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.796 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.796 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:22:16.796 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:22:16.796 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.796 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.796 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:22:16.796 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:16.796 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.796 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.796 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:22:16.796 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:22:16.796 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.796 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.797 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:22:16.797 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:16.797 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.797 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.797 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:22:16.797 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:22:16.797 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.797 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.797 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:22:16.797 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:16.797 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.797 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.797 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:22:16.798 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:22:16.798 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.798 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.798 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:22:16.798 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:16.798 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.798 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.798 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:22:16.798 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:22:16.798 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.798 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.798 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:22:16.798 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:16.798 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.799 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.799 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:22:16.799 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:22:16.799 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.799 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:22:16.799 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:22:16.799 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.799 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-22T03:22:16.799 INFO:teuthology.orchestra.run.smithi196.stdout: "type": "client_request", 2024-09-22T03:22:16.799 INFO:teuthology.orchestra.run.smithi196.stdout: "id": 18, 2024-09-22T03:22:16.799 INFO:teuthology.orchestra.run.smithi196.stdout: "detail": { 2024-09-22T03:22:16.800 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:22:16.800 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:22:16.800 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:22:15.352677+0000" 2024-09-22T03:22:16.800 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.800 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:22:16.800 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:22:16.800 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:22:15.352687+0000" 2024-09-22T03:22:16.800 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.800 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:22:16.800 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:16.800 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.800 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.801 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:22:16.801 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:22:16.801 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:22:15.459679+0000" 2024-09-22T03:22:16.801 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.801 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:22:16.801 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:16.801 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.801 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.801 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:22:16.801 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:22:16.801 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.801 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.801 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:22:16.801 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:16.802 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.802 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.802 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:22:16.802 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:22:16.802 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:22:15.460271+0000" 2024-09-22T03:22:16.802 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.802 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:22:16.802 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:16.802 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.802 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.802 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:22:16.802 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:22:16.802 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:22:15.461917+0000", 2024-09-22T03:22:16.802 INFO:teuthology.orchestra.run.smithi196.stdout: "blocker": { 2024-09-22T03:22:16.803 INFO:teuthology.orchestra.run.smithi196.stdout: "op_type": "PerShardPipeline::create_or_wait_pg", 2024-09-22T03:22:16.803 INFO:teuthology.orchestra.run.smithi196.stdout: "detail": { 2024-09-22T03:22:16.803 INFO:teuthology.orchestra.run.smithi196.stdout: "waiting": 5, 2024-09-22T03:22:16.803 INFO:teuthology.orchestra.run.smithi196.stdout: "held_by_operation_id": 17 2024-09-22T03:22:16.803 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:22:16.803 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:22:16.803 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.803 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:22:16.803 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:16.803 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.803 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.803 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:22:16.804 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:22:16.804 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.804 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.804 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:22:16.804 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:16.804 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.804 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.804 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:22:16.804 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:22:16.804 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.804 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.804 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:22:16.804 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:22:16.805 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.805 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.805 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:22:16.805 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:16.805 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.805 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.805 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:22:16.805 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:22:16.805 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.805 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.805 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:22:16.805 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:16.805 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.806 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.806 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:22:16.806 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:22:16.806 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.806 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.806 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:22:16.806 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:16.806 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.806 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.806 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:22:16.806 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:22:16.806 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.806 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.806 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:22:16.807 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:16.807 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.807 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.807 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:22:16.807 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:22:16.807 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.807 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.807 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:22:16.807 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:16.807 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.807 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.807 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:22:16.807 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:22:16.808 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.808 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.808 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:22:16.808 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:16.808 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.808 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.808 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:22:16.808 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:22:16.808 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.808 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.808 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:22:16.808 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:16.808 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.809 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.809 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:22:16.809 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:22:16.809 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.809 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.809 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:22:16.809 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:16.809 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.809 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.809 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:22:16.809 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:22:16.809 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.809 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.809 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:22:16.810 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:16.810 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.810 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.810 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:22:16.810 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:22:16.810 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.810 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.810 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:22:16.810 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:16.810 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.810 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.810 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:22:16.810 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:22:16.811 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.811 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.811 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:22:16.811 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:16.811 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.811 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.811 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:22:16.811 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:22:16.811 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.811 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.811 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:22:16.811 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:16.811 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.811 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.812 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:22:16.812 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:22:16.812 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.812 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.812 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:22:16.812 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:16.812 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.812 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.812 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:22:16.812 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:22:16.812 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.812 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:22:16.812 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:22:16.813 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.813 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-22T03:22:16.813 INFO:teuthology.orchestra.run.smithi196.stdout: "type": "client_request", 2024-09-22T03:22:16.813 INFO:teuthology.orchestra.run.smithi196.stdout: "id": 19, 2024-09-22T03:22:16.813 INFO:teuthology.orchestra.run.smithi196.stdout: "detail": { 2024-09-22T03:22:16.813 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:22:16.813 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:22:16.813 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:22:15.352882+0000" 2024-09-22T03:22:16.813 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.813 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:22:16.813 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:22:16.813 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:22:15.352886+0000" 2024-09-22T03:22:16.813 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.813 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:22:16.814 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:16.814 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.814 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.814 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:22:16.814 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:22:16.814 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:22:15.460323+0000" 2024-09-22T03:22:16.814 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.814 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:22:16.814 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:16.814 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.814 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.814 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:22:16.814 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:22:16.815 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.815 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.815 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:22:16.815 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:16.815 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.815 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.815 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:22:16.815 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:22:16.815 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:22:15.460603+0000" 2024-09-22T03:22:16.815 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.815 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:22:16.815 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:16.815 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.815 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.816 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:22:16.816 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:22:16.816 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:22:15.461952+0000", 2024-09-22T03:22:16.816 INFO:teuthology.orchestra.run.smithi196.stdout: "blocker": { 2024-09-22T03:22:16.816 INFO:teuthology.orchestra.run.smithi196.stdout: "op_type": "PerShardPipeline::create_or_wait_pg", 2024-09-22T03:22:16.816 INFO:teuthology.orchestra.run.smithi196.stdout: "detail": { 2024-09-22T03:22:16.816 INFO:teuthology.orchestra.run.smithi196.stdout: "waiting": 5, 2024-09-22T03:22:16.816 INFO:teuthology.orchestra.run.smithi196.stdout: "held_by_operation_id": 17 2024-09-22T03:22:16.816 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:22:16.816 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:22:16.816 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.816 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:22:16.816 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:16.817 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.817 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.817 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:22:16.817 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:22:16.817 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.817 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.817 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:22:16.817 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:16.817 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.817 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.817 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:22:16.817 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:22:16.817 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.818 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.818 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:22:16.818 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:22:16.818 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.818 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.818 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:22:16.818 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:16.818 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.818 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.818 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:22:16.818 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:22:16.818 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.818 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.819 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:22:16.819 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:16.819 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.819 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.819 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:22:16.819 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:22:16.819 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.819 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.819 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:22:16.819 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:16.819 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.819 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.819 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:22:16.820 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:22:16.820 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.820 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.820 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:22:16.820 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:16.820 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.820 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.820 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:22:16.820 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:22:16.820 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.820 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.820 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:22:16.820 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:16.820 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.821 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.821 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:22:16.821 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:22:16.821 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.821 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.821 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:22:16.821 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:16.821 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.821 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.821 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:22:16.821 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:22:16.821 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.821 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.822 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:22:16.822 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:16.822 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.822 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.822 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:22:16.822 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:22:16.822 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.822 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.822 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:22:16.822 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:16.822 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.822 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.822 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:22:16.822 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:22:16.823 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.823 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.823 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:22:16.823 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:16.823 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.823 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.823 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:22:16.823 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:22:16.823 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.823 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.823 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:22:16.823 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:16.823 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.823 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.824 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:22:16.824 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:22:16.824 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.824 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.824 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:22:16.824 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:16.824 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.824 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.824 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:22:16.824 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:22:16.824 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.824 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.824 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:22:16.824 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:16.825 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.825 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.825 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:22:16.825 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:22:16.825 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.825 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.825 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:22:16.825 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:16.825 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.825 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.825 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:22:16.825 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:22:16.825 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.825 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:22:16.825 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:22:16.826 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.826 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-22T03:22:16.826 INFO:teuthology.orchestra.run.smithi196.stdout: "type": "client_request", 2024-09-22T03:22:16.826 INFO:teuthology.orchestra.run.smithi196.stdout: "id": 21, 2024-09-22T03:22:16.826 INFO:teuthology.orchestra.run.smithi196.stdout: "detail": { 2024-09-22T03:22:16.826 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:22:16.826 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:22:16.826 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:22:15.353287+0000" 2024-09-22T03:22:16.826 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.826 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:22:16.826 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:22:16.826 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:22:15.353292+0000" 2024-09-22T03:22:16.826 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.826 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:22:16.826 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:16.826 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.827 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.827 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:22:16.827 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:22:16.827 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:22:15.461221+0000" 2024-09-22T03:22:16.827 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.827 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:22:16.827 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:16.827 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.827 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.827 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:22:16.827 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:22:16.827 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.827 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.827 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:22:16.828 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:16.828 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.828 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.828 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:22:16.828 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:22:16.828 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:22:15.461296+0000" 2024-09-22T03:22:16.828 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.828 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:22:16.828 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:16.828 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.828 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.828 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:22:16.828 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:22:16.828 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:22:15.462552+0000", 2024-09-22T03:22:16.829 INFO:teuthology.orchestra.run.smithi196.stdout: "blocker": { 2024-09-22T03:22:16.829 INFO:teuthology.orchestra.run.smithi196.stdout: "op_type": "PerShardPipeline::create_or_wait_pg", 2024-09-22T03:22:16.829 INFO:teuthology.orchestra.run.smithi196.stdout: "detail": { 2024-09-22T03:22:16.829 INFO:teuthology.orchestra.run.smithi196.stdout: "waiting": 5, 2024-09-22T03:22:16.829 INFO:teuthology.orchestra.run.smithi196.stdout: "held_by_operation_id": 17 2024-09-22T03:22:16.829 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:22:16.829 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:22:16.829 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.829 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:22:16.829 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:16.829 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.829 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.829 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:22:16.829 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:22:16.829 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.830 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.830 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:22:16.830 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:16.830 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.830 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.830 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:22:16.830 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:22:16.830 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.830 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.830 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:22:16.830 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:22:16.830 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.830 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.831 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:22:16.831 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:16.831 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.831 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.831 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:22:16.831 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:22:16.831 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.831 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.831 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:22:16.831 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:16.831 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.831 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.831 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:22:16.831 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:22:16.832 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.832 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.832 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:22:16.832 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:16.832 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.832 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.832 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:22:16.832 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:22:16.832 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.832 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.832 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:22:16.832 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:16.832 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.832 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.832 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:22:16.833 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:22:16.833 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.833 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.833 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:22:16.833 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:16.833 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.833 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.833 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:22:16.833 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:22:16.833 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.833 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.833 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:22:16.833 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:16.834 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.834 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.834 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:22:16.834 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:22:16.834 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.834 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.834 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:22:16.834 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:16.834 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.834 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.834 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:22:16.834 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:22:16.834 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.835 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.835 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:22:16.835 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:16.835 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.835 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.835 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:22:16.835 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:22:16.835 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.835 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.835 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:22:16.835 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:16.835 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.835 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.835 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:22:16.836 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:22:16.836 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.836 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.836 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:22:16.836 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:16.836 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.836 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.836 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:22:16.836 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:22:16.836 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.836 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.836 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:22:16.836 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:16.837 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.837 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.837 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:22:16.837 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:22:16.837 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.837 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.837 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:22:16.837 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:16.837 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.837 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.837 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:22:16.837 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:22:16.837 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.837 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.838 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:22:16.838 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:16.838 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.838 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.838 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:22:16.838 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:22:16.838 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.838 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:22:16.838 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:22:16.838 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.838 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-22T03:22:16.838 INFO:teuthology.orchestra.run.smithi196.stdout: "type": "client_request", 2024-09-22T03:22:16.838 INFO:teuthology.orchestra.run.smithi196.stdout: "id": 20, 2024-09-22T03:22:16.838 INFO:teuthology.orchestra.run.smithi196.stdout: "detail": { 2024-09-22T03:22:16.839 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:22:16.839 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:22:16.839 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:22:15.353015+0000" 2024-09-22T03:22:16.839 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.839 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:22:16.839 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:22:16.839 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:22:15.353019+0000" 2024-09-22T03:22:16.839 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.839 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:22:16.839 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:16.839 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.839 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.839 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:22:16.839 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:22:16.840 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:22:15.460811+0000" 2024-09-22T03:22:16.840 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.840 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:22:16.840 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:16.840 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.840 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.840 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:22:16.840 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:22:16.840 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.840 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.840 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:22:16.840 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:16.840 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.841 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.841 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:22:16.841 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:22:16.841 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:22:15.461165+0000" 2024-09-22T03:22:16.841 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.841 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:22:16.841 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:16.841 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.841 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.841 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:22:16.841 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:22:16.841 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:22:15.462523+0000", 2024-09-22T03:22:16.841 INFO:teuthology.orchestra.run.smithi196.stdout: "blocker": { 2024-09-22T03:22:16.841 INFO:teuthology.orchestra.run.smithi196.stdout: "op_type": "PerShardPipeline::create_or_wait_pg", 2024-09-22T03:22:16.842 INFO:teuthology.orchestra.run.smithi196.stdout: "detail": { 2024-09-22T03:22:16.842 INFO:teuthology.orchestra.run.smithi196.stdout: "waiting": 5, 2024-09-22T03:22:16.842 INFO:teuthology.orchestra.run.smithi196.stdout: "held_by_operation_id": 17 2024-09-22T03:22:16.842 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:22:16.842 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:22:16.842 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.842 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:22:16.842 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:16.842 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.842 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.842 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:22:16.842 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:22:16.842 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.842 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.843 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:22:16.843 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:16.843 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.843 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.843 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:22:16.843 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:22:16.843 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.843 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.843 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:22:16.843 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:22:16.843 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.843 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.843 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:22:16.844 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:16.844 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.844 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.844 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:22:16.844 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:22:16.844 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.844 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.844 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:22:16.844 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:16.844 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.844 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.844 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:22:16.844 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:22:16.844 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.845 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.845 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:22:16.845 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:16.845 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.845 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.845 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:22:16.845 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:22:16.845 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.845 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.845 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:22:16.845 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:16.845 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.845 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.846 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:22:16.846 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:22:16.846 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.846 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.846 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:22:16.846 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:16.846 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.846 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.846 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:22:16.846 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:22:16.846 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.846 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.846 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:22:16.847 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:16.847 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.847 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.847 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:22:16.847 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:22:16.847 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.847 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.847 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:22:16.847 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:16.847 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.847 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.847 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:22:16.847 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:22:16.848 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.848 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.848 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:22:16.848 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:16.848 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.848 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.848 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:22:16.848 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:22:16.848 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.848 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.848 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:22:16.848 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:16.848 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.849 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.849 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:22:16.849 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:22:16.849 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.849 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.849 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:22:16.849 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:16.849 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.849 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.849 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:22:16.849 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:22:16.849 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.849 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.849 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:22:16.850 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:16.850 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.850 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.850 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:22:16.850 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:22:16.850 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.850 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.850 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:22:16.850 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:16.850 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.850 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.850 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:22:16.850 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:22:16.851 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.851 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.851 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:22:16.851 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:16.851 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.851 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.851 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:22:16.851 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:22:16.851 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.851 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:22:16.851 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:22:16.851 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.851 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-09-22T03:22:16.851 INFO:teuthology.orchestra.run.smithi196.stdout: "type": "client_request", 2024-09-22T03:22:16.852 INFO:teuthology.orchestra.run.smithi196.stdout: "id": 22, 2024-09-22T03:22:16.852 INFO:teuthology.orchestra.run.smithi196.stdout: "detail": { 2024-09-22T03:22:16.852 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:22:16.852 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:22:16.852 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:22:15.353572+0000" 2024-09-22T03:22:16.852 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.852 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:22:16.852 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:22:16.852 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:22:15.353576+0000" 2024-09-22T03:22:16.852 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.852 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:22:16.852 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:16.852 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.853 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.853 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:22:16.853 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:22:16.853 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:22:15.461924+0000" 2024-09-22T03:22:16.853 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.853 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:22:16.853 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:16.853 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.853 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.853 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:22:16.853 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:22:16.853 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.853 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.853 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:22:16.854 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:16.854 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.854 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.854 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:22:16.854 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:22:16.854 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:22:15.463032+0000" 2024-09-22T03:22:16.854 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.854 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:22:16.854 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:16.854 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.854 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.854 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:22:16.854 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:22:16.855 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-09-22T03:22:15.463961+0000", 2024-09-22T03:22:16.855 INFO:teuthology.orchestra.run.smithi196.stdout: "blocker": { 2024-09-22T03:22:16.855 INFO:teuthology.orchestra.run.smithi196.stdout: "op_type": "PerShardPipeline::create_or_wait_pg", 2024-09-22T03:22:16.855 INFO:teuthology.orchestra.run.smithi196.stdout: "detail": { 2024-09-22T03:22:16.855 INFO:teuthology.orchestra.run.smithi196.stdout: "waiting": 5, 2024-09-22T03:22:16.855 INFO:teuthology.orchestra.run.smithi196.stdout: "held_by_operation_id": 17 2024-09-22T03:22:16.855 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:22:16.855 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:22:16.855 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.855 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:22:16.855 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:16.855 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.855 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.855 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:22:16.856 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:22:16.856 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.856 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.856 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:22:16.856 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:16.856 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.856 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.856 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:22:16.856 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:22:16.856 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.856 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.856 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:22:16.856 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:22:16.857 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.857 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.857 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:22:16.857 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:16.857 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.857 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.857 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:22:16.857 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:22:16.857 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.857 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.857 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:22:16.857 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:16.857 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.858 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.858 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:22:16.858 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:22:16.858 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.858 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.858 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:22:16.858 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:16.858 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.858 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.858 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:22:16.858 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:22:16.858 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.858 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.858 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:22:16.859 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:16.859 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.859 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.859 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:22:16.859 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:22:16.859 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.859 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.859 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:22:16.859 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:16.859 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.859 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.859 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:22:16.859 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:22:16.859 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.860 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.860 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:22:16.860 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:16.860 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.860 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.860 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:22:16.860 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:22:16.860 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.860 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.860 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:22:16.860 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:16.860 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.860 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.861 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:22:16.861 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:22:16.861 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.861 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.861 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:22:16.861 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:16.861 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.861 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.861 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:22:16.861 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:22:16.861 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.861 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.861 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:22:16.861 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:16.862 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.862 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.862 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:22:16.862 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:22:16.862 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.862 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.862 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:22:16.862 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:16.862 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.862 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.862 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:22:16.862 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:22:16.862 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.863 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.863 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:22:16.863 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:16.863 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.863 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.863 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:22:16.863 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:22:16.863 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.863 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.863 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:22:16.863 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:16.863 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.863 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.864 INFO:teuthology.orchestra.run.smithi196.stdout: "blocking_event": { 2024-09-22T03:22:16.864 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:22:16.864 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.864 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.864 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:22:16.864 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:16.864 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.864 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-09-22T03:22:16.864 INFO:teuthology.orchestra.run.smithi196.stdout: "time_event": { 2024-09-22T03:22:16.864 INFO:teuthology.orchestra.run.smithi196.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:22:16.864 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:16.864 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:22:16.864 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:22:16.864 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-09-22T03:22:16.865 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-09-22T03:22:16.865 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-09-22T03:22:17.073 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 63 16 1416 1400 0.173589 0 - 0.272671 2024-09-22T03:22:18.074 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 64 16 1416 1400 0.170877 0 - 0.272671 2024-09-22T03:22:19.074 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 65 16 1416 1400 0.168248 0 - 0.272671 2024-09-22T03:22:20.074 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 66 16 1416 1400 0.165698 0 - 0.272671 2024-09-22T03:22:21.074 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 67 16 1416 1400 0.163225 0 - 0.272671 2024-09-22T03:22:21.866 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [0] live_osds: [1, 3, 2] 2024-09-22T03:22:21.867 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-22T03:22:21.867 INFO:tasks.thrashosds.thrasher:Reviving osd 0 2024-09-22T03:22:21.867 INFO:tasks.ceph.osd.0:Restarting daemon 2024-09-22T03:22:21.867 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2>> /var/log/ceph/ceph-osd.0.log 2024-09-22T03:22:21.871 INFO:tasks.ceph.osd.0:Started 2024-09-22T03:22:21.871 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-22T03:22:21.981 INFO:teuthology.orchestra.run.smithi153.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-09-22T03:22:21.984 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-22T03:22:21.984 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-0, ['dump_ops_in_flight'] 2024-09-22T03:22:22.074 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 68 16 1416 1400 0.160825 0 - 0.272671 2024-09-22T03:22:23.074 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 69 16 1416 1400 0.158494 0 - 0.272671 2024-09-22T03:22:24.074 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 70 16 1416 1400 0.15623 0 - 0.272671 2024-09-22T03:22:24.980 INFO:teuthology.orchestra.run.smithi153.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-22T03:22:25.074 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 71 16 1416 1400 0.15403 0 - 0.272671 2024-09-22T03:22:25.081 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-22T03:22:25.081 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-22T03:22:26.074 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 72 16 1416 1400 0.151891 0 - 0.272671 2024-09-22T03:22:26.985 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-22T03:22:27.074 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 73 16 1792 1776 0.190045 0.101293 0.0399093 0.657481 2024-09-22T03:22:27.082 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-22T03:22:27.114 INFO:teuthology.orchestra.run.smithi153.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-09-22T03:22:27.117 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-22T03:22:27.117 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-0, ['dump_ops_in_flight'] 2024-09-22T03:22:27.226 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:22:27.223+0000 7f6e8706c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:22:27.230 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:22:27.228+0000 7f6e8706c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:22:27.233 INFO:teuthology.orchestra.run.smithi153.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-22T03:22:27.377 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-22T03:22:27.377 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-22T03:22:28.074 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 74 16 2142 2126 0.224423 2.73438 0.0304181 0.556826 2024-09-22T03:22:29.075 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 75 16 2500 2484 0.258718 2.79688 0.0391986 0.482513 2024-09-22T03:22:29.378 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-22T03:22:29.481 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:22:29.478+0000 7ff0a6b57640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:22:29.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:22:29.482+0000 7ff0a6b57640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:22:30.075 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 76 16 2500 2484 0.255314 0 - 0.482513 2024-09-22T03:22:31.075 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 77 16 2772 2756 0.279592 1.0625 0.0430333 0.446883 2024-09-22T03:22:31.551 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-22T03:22:31.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:22:31.696+0000 7f3d4d8a9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:22:31.704 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:22:31.702+0000 7f3d4d8a9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:22:32.075 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 78 16 3090 3074 0.307855 2.48438 0.0419152 0.405796 2024-09-22T03:22:32.118 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-22T03:22:32.309 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-22T03:22:32.310 INFO:teuthology.orchestra.run.smithi153.stdout: "ops_in_flight": [ 2024-09-22T03:22:32.310 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-22T03:22:32.310 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "client_request", 2024-09-22T03:22:32.310 INFO:teuthology.orchestra.run.smithi153.stdout: "id": 727, 2024-09-22T03:22:32.310 INFO:teuthology.orchestra.run.smithi153.stdout: "detail": { 2024-09-22T03:22:32.310 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.310 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:22:32.310 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.239433+0000" 2024-09-22T03:22:32.310 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.310 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.311 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:22:32.311 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.239439+0000" 2024-09-22T03:22:32.311 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.311 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.311 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.311 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.311 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.311 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.311 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:22:32.312 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.239542+0000" 2024-09-22T03:22:32.312 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.312 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.312 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.312 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.312 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.312 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.312 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:22:32.312 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.312 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.312 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.313 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.313 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.313 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.313 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.313 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:22:32.313 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.239618+0000" 2024-09-22T03:22:32.313 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.313 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.313 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.313 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.314 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.314 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.314 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:22:32.314 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.239777+0000" 2024-09-22T03:22:32.314 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.314 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.314 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.314 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.314 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.314 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.314 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:22:32.315 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.315 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.315 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.315 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.315 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.315 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.315 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.315 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:22:32.315 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.315 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.315 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.316 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:22:32.316 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.240150+0000" 2024-09-22T03:22:32.316 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.316 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.316 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.316 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.316 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.316 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.316 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:22:32.316 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.317 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.317 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.317 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.317 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.317 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.317 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.317 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:22:32.317 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.240709+0000" 2024-09-22T03:22:32.317 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.317 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.317 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.317 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.318 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.318 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.318 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:22:32.318 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.318 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.318 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.318 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.318 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.318 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.318 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.319 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:22:32.319 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.241395+0000" 2024-09-22T03:22:32.319 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.319 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.319 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.319 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.319 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.319 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.319 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:22:32.319 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.319 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.320 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.320 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.320 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.320 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.320 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.320 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:22:32.320 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.320 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.320 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.320 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.320 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.321 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.321 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.321 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:22:32.321 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.321 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.321 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.321 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.321 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.321 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.321 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.321 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:22:32.321 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.242057+0000" 2024-09-22T03:22:32.322 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.322 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.322 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.322 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.322 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.322 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.322 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:22:32.322 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.322 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.322 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.323 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.323 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.323 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.323 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.323 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:22:32.323 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.242373+0000" 2024-09-22T03:22:32.323 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.323 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.323 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.323 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.323 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.323 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.324 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:22:32.324 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.324 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.324 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.324 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.324 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.324 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.324 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.324 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:22:32.324 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.325 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.325 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.325 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.325 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.325 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.325 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.325 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:22:32.325 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.325 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:22:32.325 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:22:32.325 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.325 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-22T03:22:32.326 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "client_request", 2024-09-22T03:22:32.326 INFO:teuthology.orchestra.run.smithi153.stdout: "id": 728, 2024-09-22T03:22:32.326 INFO:teuthology.orchestra.run.smithi153.stdout: "detail": { 2024-09-22T03:22:32.326 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.326 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:22:32.326 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.244269+0000" 2024-09-22T03:22:32.326 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.326 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.326 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:22:32.326 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.244275+0000" 2024-09-22T03:22:32.326 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.326 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.327 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.327 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.327 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.327 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.327 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:22:32.327 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.244880+0000" 2024-09-22T03:22:32.327 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.327 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.327 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.327 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.327 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.328 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.328 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:22:32.328 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.328 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.328 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.328 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.328 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.328 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.328 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.328 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:22:32.328 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.246644+0000" 2024-09-22T03:22:32.328 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.329 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.329 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.329 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.329 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.329 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.329 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:22:32.329 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.247549+0000" 2024-09-22T03:22:32.329 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.329 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.329 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.330 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.330 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.330 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.330 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:22:32.330 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.330 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.330 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.330 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.330 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.330 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.331 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.331 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:22:32.331 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.331 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.331 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.331 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:22:32.331 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.247813+0000" 2024-09-22T03:22:32.331 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.331 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.331 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.331 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.331 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.332 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.332 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:22:32.332 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.332 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.332 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.332 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.332 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.332 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.332 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.332 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:22:32.333 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.249357+0000" 2024-09-22T03:22:32.333 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.333 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.333 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.333 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.333 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.333 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.333 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:22:32.333 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.333 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.334 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.334 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.334 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.334 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.334 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.334 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:22:32.334 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.249501+0000" 2024-09-22T03:22:32.334 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.334 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.334 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.334 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.335 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.335 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.335 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:22:32.335 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.335 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.335 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.335 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.335 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.335 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.335 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.335 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:22:32.336 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.336 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.336 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.336 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.336 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.336 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.336 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.336 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:22:32.336 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.336 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.336 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.337 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.337 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.337 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.337 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.337 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:22:32.337 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.250637+0000" 2024-09-22T03:22:32.337 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.337 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.337 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.337 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.338 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.338 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.338 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:22:32.338 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.338 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.338 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.338 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.338 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.338 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.338 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.338 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:22:32.338 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.251557+0000" 2024-09-22T03:22:32.339 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.339 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.339 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.339 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.339 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.339 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.339 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:22:32.339 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.339 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.339 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.339 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.340 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.340 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.340 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.340 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:22:32.340 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.340 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.340 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.340 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.340 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.340 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.340 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.341 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:22:32.341 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.341 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:22:32.341 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:22:32.341 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.341 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-22T03:22:32.341 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "client_request", 2024-09-22T03:22:32.341 INFO:teuthology.orchestra.run.smithi153.stdout: "id": 729, 2024-09-22T03:22:32.341 INFO:teuthology.orchestra.run.smithi153.stdout: "detail": { 2024-09-22T03:22:32.341 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.341 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:22:32.342 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.251313+0000" 2024-09-22T03:22:32.342 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.342 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.342 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:22:32.342 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.251324+0000" 2024-09-22T03:22:32.342 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.342 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.342 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.342 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.342 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.342 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.343 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:22:32.343 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.252057+0000" 2024-09-22T03:22:32.343 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.343 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.343 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.343 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.343 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.343 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.343 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:22:32.343 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.343 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.344 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.344 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.344 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.344 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.344 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.344 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:22:32.344 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.254320+0000" 2024-09-22T03:22:32.344 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.344 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.344 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.344 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.344 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.344 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.344 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:22:32.345 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.254525+0000" 2024-09-22T03:22:32.345 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.345 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.345 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.345 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.345 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.345 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.345 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:22:32.345 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.345 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.345 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.345 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.345 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.346 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.346 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.346 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:22:32.346 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.346 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.346 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.346 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:22:32.346 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.254920+0000" 2024-09-22T03:22:32.346 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.346 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.346 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.346 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.347 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.347 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.347 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:22:32.347 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.347 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.347 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.347 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.347 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.347 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.347 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.347 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:22:32.347 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.256755+0000" 2024-09-22T03:22:32.348 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.348 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.348 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.348 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.348 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.348 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.348 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:22:32.348 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.348 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.348 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.348 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.348 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.348 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.349 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.349 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:22:32.349 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.257060+0000" 2024-09-22T03:22:32.349 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.349 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.349 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.349 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.349 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.349 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.349 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:22:32.350 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.350 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.350 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.350 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.350 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.350 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.350 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.350 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:22:32.350 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.350 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.350 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.350 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.350 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.351 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.351 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.351 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:22:32.351 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.351 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.351 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.351 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.351 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.351 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.351 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.351 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:22:32.351 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.257317+0000" 2024-09-22T03:22:32.351 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.352 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.352 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.352 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.352 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.352 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.352 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:22:32.352 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.352 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.352 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.352 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.352 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.352 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.353 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.353 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:22:32.353 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.258742+0000" 2024-09-22T03:22:32.353 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.353 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.353 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.353 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.353 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.353 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.353 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:22:32.353 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.353 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.354 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.354 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.354 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.354 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.354 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.354 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:22:32.354 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.354 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.354 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.354 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.354 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.355 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.355 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.355 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:22:32.355 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.355 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:22:32.355 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:22:32.355 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.355 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-22T03:22:32.355 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "client_request", 2024-09-22T03:22:32.355 INFO:teuthology.orchestra.run.smithi153.stdout: "id": 730, 2024-09-22T03:22:32.355 INFO:teuthology.orchestra.run.smithi153.stdout: "detail": { 2024-09-22T03:22:32.355 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.355 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:22:32.356 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.262193+0000" 2024-09-22T03:22:32.356 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.356 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.356 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:22:32.356 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.262200+0000" 2024-09-22T03:22:32.356 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.356 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.356 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.356 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.356 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.356 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.356 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:22:32.357 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.263123+0000", 2024-09-22T03:22:32.357 INFO:teuthology.orchestra.run.smithi153.stdout: "blocker": { 2024-09-22T03:22:32.357 INFO:teuthology.orchestra.run.smithi153.stdout: "op_type": "ConnectionPipeline::await_map", 2024-09-22T03:22:32.357 INFO:teuthology.orchestra.run.smithi153.stdout: "detail": { 2024-09-22T03:22:32.357 INFO:teuthology.orchestra.run.smithi153.stdout: "waiting": 0, 2024-09-22T03:22:32.357 INFO:teuthology.orchestra.run.smithi153.stdout: "held_by_operation_id": 730 2024-09-22T03:22:32.357 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:22:32.357 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:22:32.357 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.357 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.358 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.358 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.358 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.358 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.358 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:22:32.358 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.358 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.358 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.358 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.358 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.358 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.359 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.359 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:22:32.359 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.359 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.359 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.359 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.359 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.359 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.359 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.359 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:22:32.359 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.359 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.359 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.360 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.360 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.360 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.360 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.360 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:22:32.360 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.360 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.360 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.360 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.360 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.361 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.361 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.361 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:22:32.361 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.361 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.361 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.361 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:22:32.361 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.361 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.361 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.361 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.361 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.362 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.362 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.362 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:22:32.362 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.362 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.363 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.363 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.363 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.363 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.363 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.363 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:22:32.364 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.364 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.364 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.364 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.364 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.364 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.365 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.365 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:22:32.365 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.365 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.365 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.365 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.366 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.366 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.366 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.366 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:22:32.366 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.366 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.366 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.367 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.367 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.367 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.367 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.367 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:22:32.367 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.367 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.367 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.367 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.367 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.367 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.367 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.367 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:22:32.368 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.368 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.368 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.369 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.369 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.370 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.370 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.370 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:22:32.370 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.370 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.370 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.370 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.370 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.370 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.370 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.370 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:22:32.370 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.371 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.371 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.371 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.371 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.371 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.371 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.371 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:22:32.371 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.371 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.371 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.371 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.371 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.371 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.371 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.372 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:22:32.372 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.372 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.372 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.372 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.372 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.372 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.372 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.372 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:22:32.372 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.372 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.372 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.372 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.373 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.373 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.373 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.373 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:22:32.373 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.373 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.373 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.373 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.373 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.373 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.373 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.373 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:22:32.373 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.374 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:22:32.374 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:22:32.374 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.374 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-22T03:22:32.374 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "client_request", 2024-09-22T03:22:32.374 INFO:teuthology.orchestra.run.smithi153.stdout: "id": 507, 2024-09-22T03:22:32.374 INFO:teuthology.orchestra.run.smithi153.stdout: "detail": { 2024-09-22T03:22:32.374 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.374 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:22:32.374 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.193056+0000" 2024-09-22T03:22:32.374 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.375 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.375 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:22:32.375 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.193063+0000" 2024-09-22T03:22:32.375 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.375 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.375 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.375 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.375 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.375 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.375 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:22:32.375 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.193539+0000" 2024-09-22T03:22:32.376 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.376 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.376 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.376 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.376 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.376 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.376 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:22:32.376 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.376 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.376 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.376 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.376 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.376 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.377 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.377 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:22:32.377 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.193791+0000" 2024-09-22T03:22:32.377 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.377 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.377 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.377 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.377 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.377 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.377 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:22:32.377 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.194695+0000" 2024-09-22T03:22:32.377 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.377 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.377 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.378 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.378 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.378 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.378 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:22:32.378 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.378 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.378 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.378 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.378 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.378 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.378 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.378 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:22:32.378 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.236665+0000" 2024-09-22T03:22:32.379 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.379 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.379 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:22:32.379 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.194951+0000" 2024-09-22T03:22:32.379 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.379 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.379 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.379 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.379 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.379 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.379 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:22:32.379 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.379 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.379 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.379 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.380 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.380 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.380 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.380 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:22:32.380 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.195067+0000" 2024-09-22T03:22:32.380 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.380 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.380 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.380 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.380 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.380 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.380 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:22:32.380 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.380 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.381 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.381 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.381 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.381 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.381 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.381 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:22:32.381 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.195979+0000" 2024-09-22T03:22:32.381 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.381 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.381 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.381 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.381 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.381 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.381 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:22:32.382 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.382 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.382 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.382 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.382 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.382 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.382 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.382 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:22:32.382 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.382 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.382 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.382 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.383 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.383 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.383 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.383 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:22:32.383 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.383 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.383 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.383 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.383 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.383 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.383 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.384 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:22:32.384 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.196149+0000" 2024-09-22T03:22:32.384 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.384 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.384 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.384 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.384 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.384 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.384 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:22:32.384 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.384 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.384 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.384 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.385 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.385 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.385 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.385 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:22:32.385 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.196778+0000" 2024-09-22T03:22:32.385 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.385 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.385 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.385 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.385 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.385 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.386 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:22:32.386 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.386 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.386 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.386 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.386 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.386 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.386 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.386 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:22:32.386 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.236235+0000" 2024-09-22T03:22:32.386 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.386 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.386 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.387 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.387 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.387 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.387 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:22:32.387 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.387 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:22:32.387 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:22:32.387 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.387 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-22T03:22:32.387 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "client_request", 2024-09-22T03:22:32.387 INFO:teuthology.orchestra.run.smithi153.stdout: "id": 556, 2024-09-22T03:22:32.387 INFO:teuthology.orchestra.run.smithi153.stdout: "detail": { 2024-09-22T03:22:32.387 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.388 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:22:32.388 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.386310+0000" 2024-09-22T03:22:32.388 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.388 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.388 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:22:32.388 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.386340+0000" 2024-09-22T03:22:32.388 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.388 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.388 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.388 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.388 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.388 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.388 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:22:32.388 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.386434+0000" 2024-09-22T03:22:32.389 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.389 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.389 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.389 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.389 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.389 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.389 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:22:32.389 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.389 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.389 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.389 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.389 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.389 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.390 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.390 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:22:32.390 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.386485+0000" 2024-09-22T03:22:32.390 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.390 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.390 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.390 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.390 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.390 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.390 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:22:32.390 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.386542+0000" 2024-09-22T03:22:32.390 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.390 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.390 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.391 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.391 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.391 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.391 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:22:32.391 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.391 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.391 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.391 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.391 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.391 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.391 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.391 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:22:32.391 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.426531+0000" 2024-09-22T03:22:32.392 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.392 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.392 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:22:32.392 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.386750+0000" 2024-09-22T03:22:32.392 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.392 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.392 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.392 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.392 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.392 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.392 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:22:32.392 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.392 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.392 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.393 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.393 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.393 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.393 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.393 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:22:32.393 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.386855+0000" 2024-09-22T03:22:32.393 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.393 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.393 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.393 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.393 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.393 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.393 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:22:32.394 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.394 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.394 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.394 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.394 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.394 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.394 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.394 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:22:32.394 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.386977+0000" 2024-09-22T03:22:32.394 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.394 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.394 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.394 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.395 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.395 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.395 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:22:32.395 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.395 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.395 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.395 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.395 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.395 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.395 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.395 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:22:32.395 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.395 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.396 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.396 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.396 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.396 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.396 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.396 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:22:32.396 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.396 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.396 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.396 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.396 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.396 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.396 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.396 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:22:32.397 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.387127+0000" 2024-09-22T03:22:32.397 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.397 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.397 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.397 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.397 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.397 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.397 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:22:32.397 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.397 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.397 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.397 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.397 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.397 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.398 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.398 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:22:32.398 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.388481+0000" 2024-09-22T03:22:32.398 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.398 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.398 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.398 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.398 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.398 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.398 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:22:32.398 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.398 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.398 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.398 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.399 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.399 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.399 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.399 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:22:32.399 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.424166+0000" 2024-09-22T03:22:32.399 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.399 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.399 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.399 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.399 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.399 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.399 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:22:32.399 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.400 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:22:32.400 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:22:32.400 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.400 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-22T03:22:32.400 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "client_request", 2024-09-22T03:22:32.400 INFO:teuthology.orchestra.run.smithi153.stdout: "id": 557, 2024-09-22T03:22:32.400 INFO:teuthology.orchestra.run.smithi153.stdout: "detail": { 2024-09-22T03:22:32.400 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.400 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:22:32.400 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.390950+0000" 2024-09-22T03:22:32.400 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.400 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.400 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:22:32.400 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.390966+0000" 2024-09-22T03:22:32.400 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.401 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.401 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.401 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.401 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.401 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.401 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:22:32.401 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.391054+0000" 2024-09-22T03:22:32.401 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.401 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.401 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.401 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.401 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.401 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.401 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:22:32.402 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.402 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.402 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.402 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.402 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.402 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.402 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.402 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:22:32.402 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.391341+0000" 2024-09-22T03:22:32.402 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.402 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.402 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.402 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.403 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.403 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.403 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:22:32.403 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.391416+0000" 2024-09-22T03:22:32.403 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.403 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.403 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.403 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.403 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.403 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.403 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:22:32.403 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.403 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.403 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.404 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.404 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.404 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.404 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.404 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:22:32.404 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.441978+0000" 2024-09-22T03:22:32.404 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.404 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.404 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:22:32.404 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.391921+0000" 2024-09-22T03:22:32.404 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.404 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.404 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.405 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.405 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.405 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.405 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:22:32.405 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.405 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.405 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.405 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.405 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.405 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.405 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.405 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:22:32.405 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.392079+0000" 2024-09-22T03:22:32.406 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.406 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.406 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.406 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.406 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.406 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.406 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:22:32.406 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.406 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.406 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.406 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.406 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.407 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.407 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.407 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:22:32.407 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.392238+0000" 2024-09-22T03:22:32.407 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.407 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.407 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.407 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.407 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.407 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.407 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:22:32.407 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.407 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.408 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.408 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.408 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.408 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.408 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.408 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:22:32.408 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.408 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.408 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.408 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.408 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.408 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.408 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.408 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:22:32.409 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.409 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.409 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.409 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.409 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.409 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.409 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.409 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:22:32.409 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.392442+0000" 2024-09-22T03:22:32.409 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.409 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.409 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.409 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.410 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.410 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.410 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:22:32.410 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.410 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.410 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.410 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.410 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.410 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.410 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.410 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:22:32.410 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.393108+0000" 2024-09-22T03:22:32.410 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.411 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.411 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.411 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.411 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.411 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.411 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:22:32.411 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.411 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.411 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.411 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.411 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.411 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.411 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.411 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:22:32.412 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.440630+0000" 2024-09-22T03:22:32.412 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.412 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.412 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.412 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.412 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.412 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.412 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:22:32.412 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.412 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:22:32.412 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:22:32.412 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.412 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-22T03:22:32.412 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "client_request", 2024-09-22T03:22:32.413 INFO:teuthology.orchestra.run.smithi153.stdout: "id": 558, 2024-09-22T03:22:32.413 INFO:teuthology.orchestra.run.smithi153.stdout: "detail": { 2024-09-22T03:22:32.413 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.413 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:22:32.413 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.393075+0000" 2024-09-22T03:22:32.413 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.413 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.413 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:22:32.413 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.393081+0000" 2024-09-22T03:22:32.413 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.413 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.413 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.413 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.413 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.414 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.414 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:22:32.414 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.393170+0000" 2024-09-22T03:22:32.414 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.414 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.414 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.414 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.414 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.414 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.414 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:22:32.414 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.415 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.415 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.415 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.415 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.415 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.415 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.415 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:22:32.415 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.393498+0000" 2024-09-22T03:22:32.415 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.415 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.415 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.415 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.415 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.416 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.416 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:22:32.416 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.394074+0000" 2024-09-22T03:22:32.416 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.416 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.416 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.416 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.416 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.416 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.416 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:22:32.416 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.416 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.416 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.416 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.416 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.417 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.417 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.417 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:22:32.417 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.445126+0000" 2024-09-22T03:22:32.417 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.417 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.417 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:22:32.417 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.395711+0000" 2024-09-22T03:22:32.417 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.417 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.417 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.417 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.417 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.418 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.418 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:22:32.418 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.418 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.418 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.418 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.418 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.418 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.418 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.418 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:22:32.418 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.396040+0000" 2024-09-22T03:22:32.418 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.418 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.419 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.419 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.419 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.419 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.419 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:22:32.419 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.419 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.419 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.419 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.419 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.419 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.419 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.419 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:22:32.419 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.396262+0000" 2024-09-22T03:22:32.420 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.420 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.420 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.420 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.420 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.420 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.420 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:22:32.420 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.420 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.420 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.420 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.420 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.420 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.420 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.421 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:22:32.421 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.421 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.421 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.421 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.421 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.421 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.421 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.421 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:22:32.421 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.421 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.421 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.421 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.421 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.422 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.422 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.422 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:22:32.422 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.396459+0000" 2024-09-22T03:22:32.422 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.422 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.422 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.422 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.422 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.422 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.422 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:22:32.422 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.422 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.422 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.423 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.423 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.423 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.423 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.423 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:22:32.423 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.397245+0000" 2024-09-22T03:22:32.423 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.423 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.423 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.423 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.423 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.423 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.423 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:22:32.423 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.424 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.424 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.424 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.424 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.424 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.424 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.424 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:22:32.424 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.443848+0000" 2024-09-22T03:22:32.424 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.424 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.424 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.424 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.424 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.424 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.425 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:22:32.425 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.425 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:22:32.425 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:22:32.425 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.425 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-22T03:22:32.425 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "client_request", 2024-09-22T03:22:32.425 INFO:teuthology.orchestra.run.smithi153.stdout: "id": 559, 2024-09-22T03:22:32.425 INFO:teuthology.orchestra.run.smithi153.stdout: "detail": { 2024-09-22T03:22:32.425 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.425 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:22:32.425 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.399855+0000" 2024-09-22T03:22:32.425 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.426 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.426 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:22:32.426 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.399865+0000" 2024-09-22T03:22:32.426 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.426 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.426 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.426 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.426 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.426 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.426 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:22:32.426 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.400007+0000" 2024-09-22T03:22:32.426 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.426 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.427 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.427 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.427 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.427 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.427 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:22:32.427 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.427 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.427 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.427 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.427 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.427 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.427 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.427 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:22:32.427 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.400167+0000" 2024-09-22T03:22:32.428 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.428 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.428 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.428 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.428 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.428 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.428 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:22:32.428 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.400305+0000" 2024-09-22T03:22:32.428 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.428 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.428 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.428 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.428 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.428 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.429 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:22:32.429 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.429 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.429 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.429 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.429 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.429 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.429 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.429 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:22:32.429 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.464250+0000" 2024-09-22T03:22:32.429 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.429 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.429 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:22:32.430 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.400514+0000" 2024-09-22T03:22:32.430 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.430 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.430 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.430 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.430 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.430 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.430 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:22:32.430 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.430 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.430 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.430 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.430 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.430 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.431 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.431 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:22:32.431 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.400647+0000" 2024-09-22T03:22:32.431 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.431 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.431 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.431 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.431 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.431 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.431 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:22:32.431 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.431 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.431 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.431 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.432 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.432 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.432 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.432 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:22:32.432 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.400755+0000" 2024-09-22T03:22:32.432 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.432 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.432 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.432 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.432 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.432 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.432 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:22:32.432 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.432 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.433 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.433 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.433 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.433 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.433 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.433 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:22:32.433 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.433 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.433 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.433 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.433 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.433 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.433 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.433 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:22:32.434 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.434 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.434 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.434 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.434 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.434 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.434 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.434 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:22:32.434 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.401287+0000" 2024-09-22T03:22:32.434 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.434 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.434 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.434 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.435 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.435 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.435 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:22:32.435 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.435 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.435 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.435 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.435 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.435 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.435 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.435 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:22:32.435 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.401595+0000" 2024-09-22T03:22:32.435 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.435 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.436 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.436 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.436 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.436 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.436 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:22:32.436 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.436 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.436 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.436 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.436 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.436 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.436 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.436 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:22:32.437 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.462782+0000" 2024-09-22T03:22:32.437 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.437 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.437 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.437 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.437 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.437 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.437 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:22:32.437 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.437 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:22:32.437 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:22:32.437 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.437 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-22T03:22:32.437 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "client_request", 2024-09-22T03:22:32.438 INFO:teuthology.orchestra.run.smithi153.stdout: "id": 560, 2024-09-22T03:22:32.438 INFO:teuthology.orchestra.run.smithi153.stdout: "detail": { 2024-09-22T03:22:32.438 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.438 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:22:32.438 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.401835+0000" 2024-09-22T03:22:32.438 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.438 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.438 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:22:32.438 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.401841+0000" 2024-09-22T03:22:32.438 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.438 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.438 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.438 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.438 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.439 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.439 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:22:32.439 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.403019+0000" 2024-09-22T03:22:32.439 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.439 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.439 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.439 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.439 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.439 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.439 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:22:32.439 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.439 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.439 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.439 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.440 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.440 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.440 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.440 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:22:32.440 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.404098+0000" 2024-09-22T03:22:32.440 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.440 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.440 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.440 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.440 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.440 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.440 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:22:32.440 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.404530+0000" 2024-09-22T03:22:32.440 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.441 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.441 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.441 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.441 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.441 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.441 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:22:32.441 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.441 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.441 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.441 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.441 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.441 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.441 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.441 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:22:32.442 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.467005+0000" 2024-09-22T03:22:32.442 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.442 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.442 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:22:32.442 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.405204+0000" 2024-09-22T03:22:32.442 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.442 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.442 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.442 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.442 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.442 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.442 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:22:32.442 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.443 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.443 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.443 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.443 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.443 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.443 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.443 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:22:32.443 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.405885+0000" 2024-09-22T03:22:32.443 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.443 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.443 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.443 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.443 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.444 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.444 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:22:32.444 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.444 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.444 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.444 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.444 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.444 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.444 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.444 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:22:32.444 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.407507+0000" 2024-09-22T03:22:32.444 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.444 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.444 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.445 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.445 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.445 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.445 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:22:32.445 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.445 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.445 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.445 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.445 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.445 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.445 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.445 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:22:32.445 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.445 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.446 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.446 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.446 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.446 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.446 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.446 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:22:32.446 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.446 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.446 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.446 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.446 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.446 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.446 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.446 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:22:32.447 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.407836+0000" 2024-09-22T03:22:32.447 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.447 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.447 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.447 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.447 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.447 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.447 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:22:32.447 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.447 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.447 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.447 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.447 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.447 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.447 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.448 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:22:32.448 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.408955+0000" 2024-09-22T03:22:32.448 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.448 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.448 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.448 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.448 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.448 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.448 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:22:32.448 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.448 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.448 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.448 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.448 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.449 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.449 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.449 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:22:32.449 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.466416+0000" 2024-09-22T03:22:32.449 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.449 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.449 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.449 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.449 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.449 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.449 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:22:32.449 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.449 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:22:32.449 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:22:32.449 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.450 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-22T03:22:32.450 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "client_request", 2024-09-22T03:22:32.450 INFO:teuthology.orchestra.run.smithi153.stdout: "id": 561, 2024-09-22T03:22:32.450 INFO:teuthology.orchestra.run.smithi153.stdout: "detail": { 2024-09-22T03:22:32.450 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.450 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:22:32.450 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.412711+0000" 2024-09-22T03:22:32.450 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.450 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.450 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:22:32.450 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.412722+0000" 2024-09-22T03:22:32.450 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.450 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.451 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.451 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.451 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.451 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.451 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:22:32.451 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.412852+0000" 2024-09-22T03:22:32.451 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.451 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.451 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.451 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.451 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.451 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.451 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:22:32.451 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.451 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.452 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.452 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.452 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.452 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.452 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.452 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:22:32.452 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.412962+0000" 2024-09-22T03:22:32.452 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.452 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.452 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.452 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.452 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.453 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.453 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:22:32.453 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.414086+0000" 2024-09-22T03:22:32.453 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.453 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.453 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.453 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.453 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.453 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.453 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:22:32.453 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.453 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.453 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.454 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.454 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.454 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.454 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.454 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:22:32.454 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.478949+0000" 2024-09-22T03:22:32.454 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.454 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.454 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:22:32.454 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.414486+0000" 2024-09-22T03:22:32.454 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.454 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.454 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.455 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.455 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.455 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.455 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:22:32.455 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.455 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.455 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.455 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.455 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.455 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.455 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.455 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:22:32.455 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.415008+0000" 2024-09-22T03:22:32.456 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.456 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.456 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.456 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.456 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.456 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.456 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:22:32.456 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.456 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.456 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.456 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.456 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.456 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.456 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.457 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:22:32.457 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.415516+0000" 2024-09-22T03:22:32.457 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.457 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.457 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.457 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.457 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.457 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.457 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:22:32.457 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.457 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.457 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.457 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.458 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.458 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.458 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.458 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:22:32.458 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.458 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.458 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.458 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.458 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.458 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.458 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.458 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:22:32.459 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.459 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.459 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.459 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.459 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.459 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.459 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.459 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:22:32.459 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.416621+0000" 2024-09-22T03:22:32.459 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.459 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.459 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.460 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.460 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.460 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.460 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:22:32.460 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.460 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.460 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.460 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.460 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.460 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.460 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.460 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:22:32.460 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.417190+0000" 2024-09-22T03:22:32.461 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.461 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.461 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.461 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.461 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.461 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.461 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:22:32.461 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.461 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.461 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.461 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.461 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.462 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.462 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.462 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:22:32.462 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.476791+0000" 2024-09-22T03:22:32.462 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.462 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.462 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.462 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.462 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.462 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.462 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:22:32.462 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.462 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:22:32.463 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:22:32.463 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.463 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-22T03:22:32.463 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "client_request", 2024-09-22T03:22:32.463 INFO:teuthology.orchestra.run.smithi153.stdout: "id": 562, 2024-09-22T03:22:32.463 INFO:teuthology.orchestra.run.smithi153.stdout: "detail": { 2024-09-22T03:22:32.463 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.463 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:22:32.463 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.417508+0000" 2024-09-22T03:22:32.463 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.463 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.463 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:22:32.463 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.417514+0000" 2024-09-22T03:22:32.463 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.464 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.464 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.464 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.464 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.464 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.464 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:22:32.464 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.419449+0000" 2024-09-22T03:22:32.464 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.464 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.464 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.464 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.464 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.464 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.465 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:22:32.465 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.465 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.465 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.465 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.465 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.465 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.465 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.465 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:22:32.465 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.419634+0000" 2024-09-22T03:22:32.465 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.465 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.465 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.465 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.466 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.466 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.466 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:22:32.466 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.420410+0000" 2024-09-22T03:22:32.466 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.466 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.466 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.466 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.466 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.466 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.466 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:22:32.466 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.466 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.466 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.467 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.467 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.467 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.467 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.467 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:22:32.467 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.488217+0000" 2024-09-22T03:22:32.467 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.467 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.467 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:22:32.467 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.420702+0000" 2024-09-22T03:22:32.467 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.467 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.467 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.467 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.467 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.467 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.468 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:22:32.468 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.468 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.468 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.468 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.468 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.468 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.468 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.468 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:22:32.468 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.420881+0000" 2024-09-22T03:22:32.468 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.468 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.468 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.468 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.469 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.469 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.469 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:22:32.469 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.469 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.469 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.469 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.469 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.469 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.469 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.469 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:22:32.469 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.421450+0000" 2024-09-22T03:22:32.469 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.469 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.469 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.469 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.470 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.470 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.470 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:22:32.470 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.470 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.470 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.470 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.470 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.470 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.470 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.470 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:22:32.470 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.470 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.470 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.471 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.471 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.471 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.471 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.471 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:22:32.471 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.471 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.471 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.471 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.471 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.471 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.471 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.471 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:22:32.471 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.421715+0000" 2024-09-22T03:22:32.472 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.472 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.472 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.472 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.472 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.472 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.472 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:22:32.472 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.472 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.472 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.472 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.472 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.472 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.472 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.473 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:22:32.473 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.422603+0000" 2024-09-22T03:22:32.473 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.473 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.473 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.473 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.473 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.473 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.473 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:22:32.473 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.473 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.473 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.473 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.473 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.473 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.474 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.474 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:22:32.474 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.486737+0000" 2024-09-22T03:22:32.474 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.474 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.474 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.474 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.474 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.474 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.474 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:22:32.474 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.474 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:22:32.474 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:22:32.474 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.475 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-22T03:22:32.475 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "client_request", 2024-09-22T03:22:32.475 INFO:teuthology.orchestra.run.smithi153.stdout: "id": 563, 2024-09-22T03:22:32.475 INFO:teuthology.orchestra.run.smithi153.stdout: "detail": { 2024-09-22T03:22:32.475 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.475 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:22:32.475 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.426044+0000" 2024-09-22T03:22:32.475 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.475 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.475 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:22:32.475 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.426056+0000" 2024-09-22T03:22:32.475 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.475 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.475 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.476 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.476 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.476 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.476 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:22:32.476 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.426300+0000" 2024-09-22T03:22:32.476 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.476 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.476 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.476 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.476 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.476 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.476 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:22:32.476 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.476 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.476 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.477 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.477 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.477 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.477 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.477 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:22:32.477 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.426637+0000" 2024-09-22T03:22:32.477 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.477 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.477 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.477 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.477 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.477 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.477 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:22:32.477 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.426787+0000" 2024-09-22T03:22:32.478 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.478 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.478 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.478 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.478 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.478 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.478 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:22:32.478 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.478 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.478 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.478 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.478 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.478 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.478 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.479 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:22:32.479 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.495982+0000" 2024-09-22T03:22:32.479 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.479 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.479 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:22:32.479 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.427012+0000" 2024-09-22T03:22:32.479 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.479 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.479 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.479 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.479 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.479 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.480 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:22:32.480 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.480 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.480 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.480 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.480 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.480 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.480 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.480 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:22:32.480 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.427128+0000" 2024-09-22T03:22:32.480 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.480 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.480 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.480 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.480 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.481 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.481 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:22:32.481 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.481 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.481 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.481 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.481 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.481 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.481 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.481 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:22:32.481 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.427901+0000" 2024-09-22T03:22:32.481 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.481 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.482 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.482 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.482 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.482 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.482 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:22:32.482 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.482 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.482 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.482 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.482 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.482 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.482 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.482 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:22:32.483 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.483 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.483 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.483 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.483 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.483 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.483 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.483 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:22:32.483 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.483 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.483 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.483 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.483 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.484 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.484 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.484 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:22:32.484 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.428329+0000" 2024-09-22T03:22:32.484 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.484 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.484 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.484 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.484 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.484 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.484 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:22:32.484 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.484 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.484 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.484 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.485 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.485 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.485 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.485 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:22:32.485 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.431311+0000" 2024-09-22T03:22:32.485 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.485 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.485 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.485 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.485 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.485 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.485 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:22:32.485 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.486 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.486 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.486 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.486 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.486 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.486 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.486 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:22:32.486 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.495274+0000" 2024-09-22T03:22:32.486 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.486 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.486 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.486 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.486 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.486 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.487 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:22:32.487 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.487 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:22:32.487 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:22:32.487 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.487 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-22T03:22:32.487 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "client_request", 2024-09-22T03:22:32.488 INFO:teuthology.orchestra.run.smithi153.stdout: "id": 581, 2024-09-22T03:22:32.488 INFO:teuthology.orchestra.run.smithi153.stdout: "detail": { 2024-09-22T03:22:32.488 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.488 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:22:32.489 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.498007+0000" 2024-09-22T03:22:32.489 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.489 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.489 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:22:32.489 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.498015+0000" 2024-09-22T03:22:32.489 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.489 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.490 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.490 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.490 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.490 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.490 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:22:32.491 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.498158+0000" 2024-09-22T03:22:32.491 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.491 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.491 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.492 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.492 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.492 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.492 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:22:32.492 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.492 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.493 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.493 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.493 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.493 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.493 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.493 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:22:32.493 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.498235+0000" 2024-09-22T03:22:32.494 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.494 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.494 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.495 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.495 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.495 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.495 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:22:32.496 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.498315+0000" 2024-09-22T03:22:32.496 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.496 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.496 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.496 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.496 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.497 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.497 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:22:32.497 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.497 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.497 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.497 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.498 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.498 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.498 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.498 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:22:32.498 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.522243+0000" 2024-09-22T03:22:32.498 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.499 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.499 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:22:32.499 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.498616+0000" 2024-09-22T03:22:32.499 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.499 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.499 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.499 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.499 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.500 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.500 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:22:32.500 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.500 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.500 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.501 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.501 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.501 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.501 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.501 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:22:32.502 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.498844+0000" 2024-09-22T03:22:32.502 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.502 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.502 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.502 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.503 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.503 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.503 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:22:32.503 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.503 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.503 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.503 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.503 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.503 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.504 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.504 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:22:32.504 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.499387+0000" 2024-09-22T03:22:32.504 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.504 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.504 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.504 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.504 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.504 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.504 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:22:32.504 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.504 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.504 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.505 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.505 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.505 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.505 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.505 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:22:32.505 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.505 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.505 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.505 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.505 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.505 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.505 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.505 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:22:32.505 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.506 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.506 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.506 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.506 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.506 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.506 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.506 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:22:32.506 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.499591+0000" 2024-09-22T03:22:32.506 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.506 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.506 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.506 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.506 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.506 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.507 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:22:32.507 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.507 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.507 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.507 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.507 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.507 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.507 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.507 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:22:32.507 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.502237+0000" 2024-09-22T03:22:32.507 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.507 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.507 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.508 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.508 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.508 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.508 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:22:32.508 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.508 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.508 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.508 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.508 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.508 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.508 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.508 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:22:32.508 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.518649+0000" 2024-09-22T03:22:32.509 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.509 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.509 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.509 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.509 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.509 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.509 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:22:32.509 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.509 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:22:32.509 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:22:32.509 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.509 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-22T03:22:32.509 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "client_request", 2024-09-22T03:22:32.510 INFO:teuthology.orchestra.run.smithi153.stdout: "id": 582, 2024-09-22T03:22:32.510 INFO:teuthology.orchestra.run.smithi153.stdout: "detail": { 2024-09-22T03:22:32.510 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.510 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:22:32.510 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.505902+0000" 2024-09-22T03:22:32.510 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.510 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.510 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:22:32.510 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.505913+0000" 2024-09-22T03:22:32.510 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.510 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.510 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.510 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.510 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.511 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.511 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:22:32.511 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.507358+0000" 2024-09-22T03:22:32.511 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.511 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.511 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.511 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.511 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.511 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.511 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:22:32.511 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.511 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.511 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.512 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.512 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.512 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.512 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.512 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:22:32.512 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.507512+0000" 2024-09-22T03:22:32.512 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.512 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.512 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.512 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.512 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.512 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.512 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:22:32.512 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.507674+0000" 2024-09-22T03:22:32.513 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.513 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.513 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.513 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.513 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.513 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.513 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:22:32.513 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.513 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.513 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.513 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.513 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.513 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.513 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.514 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:22:32.514 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.533114+0000" 2024-09-22T03:22:32.514 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.514 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.514 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:22:32.514 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.509074+0000" 2024-09-22T03:22:32.514 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.514 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.514 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.514 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.514 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.514 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.514 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:22:32.515 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.515 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.515 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.515 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.515 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.515 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.515 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.515 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:22:32.515 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.509292+0000" 2024-09-22T03:22:32.515 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.515 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.515 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.515 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.515 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.516 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.516 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:22:32.516 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.516 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.516 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.516 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.516 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.516 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.516 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.516 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:22:32.516 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.509549+0000" 2024-09-22T03:22:32.516 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.516 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.516 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.517 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.517 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.517 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.517 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:22:32.517 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.517 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.517 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.517 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.517 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.517 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.517 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.517 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:22:32.517 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.518 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.518 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.518 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.518 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.518 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.518 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.518 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:22:32.518 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.518 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.518 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.518 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.518 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.518 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.518 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.519 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:22:32.519 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.510601+0000" 2024-09-22T03:22:32.519 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.519 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.519 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.519 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.519 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.519 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.519 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:22:32.519 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.519 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.519 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.519 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.519 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.520 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.520 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.520 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:22:32.520 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.511859+0000" 2024-09-22T03:22:32.520 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.520 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.520 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.520 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.520 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.520 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.520 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:22:32.520 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.520 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.521 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.521 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.521 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.521 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.521 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.521 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:22:32.521 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.531039+0000" 2024-09-22T03:22:32.521 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.521 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.521 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.521 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.521 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.521 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.521 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:22:32.522 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.522 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:22:32.522 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:22:32.522 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.522 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-22T03:22:32.522 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "client_request", 2024-09-22T03:22:32.522 INFO:teuthology.orchestra.run.smithi153.stdout: "id": 584, 2024-09-22T03:22:32.522 INFO:teuthology.orchestra.run.smithi153.stdout: "detail": { 2024-09-22T03:22:32.522 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.522 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:22:32.522 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.516446+0000" 2024-09-22T03:22:32.522 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.522 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.523 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:22:32.523 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.516459+0000" 2024-09-22T03:22:32.523 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.523 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.523 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.523 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.523 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.523 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.523 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:22:32.523 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.516684+0000" 2024-09-22T03:22:32.523 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.523 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.523 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.524 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.524 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.524 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.524 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:22:32.524 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.524 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.524 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.524 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.524 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.524 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.524 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.524 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:22:32.524 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.516819+0000" 2024-09-22T03:22:32.524 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.525 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.525 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.525 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.525 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.525 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.525 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:22:32.525 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.519914+0000" 2024-09-22T03:22:32.525 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.525 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.525 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.525 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.525 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.525 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.526 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:22:32.526 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.526 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.526 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.526 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.526 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.526 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.526 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.526 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:22:32.526 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.538988+0000" 2024-09-22T03:22:32.526 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.526 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.526 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:22:32.526 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.520517+0000" 2024-09-22T03:22:32.527 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.527 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.527 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.527 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.527 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.527 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.527 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:22:32.527 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.527 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.527 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.527 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.527 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.527 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.528 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.528 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:22:32.528 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.520937+0000" 2024-09-22T03:22:32.528 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.528 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.528 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.528 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.528 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.528 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.528 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:22:32.528 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.528 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.528 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.528 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.529 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.529 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.529 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.529 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:22:32.529 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.522125+0000" 2024-09-22T03:22:32.529 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.529 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.529 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.529 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.529 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.529 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.529 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:22:32.529 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.529 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.530 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.530 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.530 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.530 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.530 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.530 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:22:32.530 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.530 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.530 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.530 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.530 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.530 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.530 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.531 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:22:32.531 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.531 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.531 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.531 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.531 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.531 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.531 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.531 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:22:32.531 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.522972+0000" 2024-09-22T03:22:32.531 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.531 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.531 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.532 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.532 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.532 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.532 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:22:32.532 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.532 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.532 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.532 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.532 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.532 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.532 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.532 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:22:32.532 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.523801+0000" 2024-09-22T03:22:32.532 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.533 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.533 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.533 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.533 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.533 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.533 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:22:32.533 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.533 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.533 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.533 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.533 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.533 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.533 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.533 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:22:32.534 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.537849+0000" 2024-09-22T03:22:32.534 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.534 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.534 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.534 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.534 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.534 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.534 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:22:32.534 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.534 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:22:32.534 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:22:32.534 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.534 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-22T03:22:32.534 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "client_request", 2024-09-22T03:22:32.535 INFO:teuthology.orchestra.run.smithi153.stdout: "id": 586, 2024-09-22T03:22:32.535 INFO:teuthology.orchestra.run.smithi153.stdout: "detail": { 2024-09-22T03:22:32.535 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.535 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:22:32.535 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.523174+0000" 2024-09-22T03:22:32.535 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.535 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.535 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:22:32.535 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.523184+0000" 2024-09-22T03:22:32.535 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.535 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.535 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.535 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.536 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.536 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.536 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:22:32.536 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.523786+0000" 2024-09-22T03:22:32.536 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.536 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.536 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.536 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.536 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.536 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.536 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:22:32.536 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.536 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.536 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.537 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.537 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.537 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.537 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.537 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:22:32.537 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.524376+0000" 2024-09-22T03:22:32.537 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.537 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.537 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.537 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.537 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.537 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.537 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:22:32.538 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.524773+0000" 2024-09-22T03:22:32.538 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.538 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.538 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.538 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.538 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.538 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.538 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:22:32.538 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.538 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.538 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.538 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.538 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.538 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.539 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.539 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:22:32.539 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.546453+0000" 2024-09-22T03:22:32.539 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.539 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.539 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:22:32.539 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.526346+0000" 2024-09-22T03:22:32.539 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.539 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.539 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.539 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.539 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.539 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.539 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:22:32.540 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.540 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.540 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.540 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.540 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.540 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.540 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.540 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:22:32.540 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.526584+0000" 2024-09-22T03:22:32.540 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.540 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.540 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.540 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.540 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.541 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.541 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:22:32.541 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.541 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.541 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.541 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.541 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.541 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.541 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.541 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:22:32.541 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.527568+0000" 2024-09-22T03:22:32.541 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.541 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.541 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.542 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.542 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.542 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.542 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:22:32.542 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.542 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.542 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.542 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.542 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.542 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.542 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.542 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:22:32.542 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.542 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.543 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.543 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.543 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.543 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.543 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.543 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:22:32.543 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.543 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.543 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.543 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.543 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.543 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.543 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.544 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:22:32.544 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.528976+0000" 2024-09-22T03:22:32.544 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.544 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.544 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.544 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.544 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.544 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.544 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:22:32.544 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.544 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.544 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.544 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.544 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.545 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.545 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.545 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:22:32.545 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.529362+0000" 2024-09-22T03:22:32.545 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.545 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.545 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.545 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.545 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.545 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.545 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:22:32.545 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.545 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.545 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.546 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.546 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.546 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.546 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.546 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:22:32.546 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.545780+0000" 2024-09-22T03:22:32.546 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.546 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.546 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.546 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.546 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.546 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.546 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:22:32.547 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.547 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:22:32.547 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:22:32.547 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.547 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-22T03:22:32.547 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "client_request", 2024-09-22T03:22:32.547 INFO:teuthology.orchestra.run.smithi153.stdout: "id": 588, 2024-09-22T03:22:32.547 INFO:teuthology.orchestra.run.smithi153.stdout: "detail": { 2024-09-22T03:22:32.547 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.547 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:22:32.547 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.534238+0000" 2024-09-22T03:22:32.547 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.547 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.547 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:22:32.548 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.534250+0000" 2024-09-22T03:22:32.548 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.548 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.548 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.548 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.548 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.548 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.548 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:22:32.548 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.535700+0000" 2024-09-22T03:22:32.548 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.548 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.549 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.549 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.549 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.549 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.549 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:22:32.549 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.549 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.549 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.549 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.549 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.550 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.550 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.550 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:22:32.550 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.535974+0000" 2024-09-22T03:22:32.550 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.550 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.550 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.550 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.550 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.550 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.550 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:22:32.551 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.536129+0000" 2024-09-22T03:22:32.551 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.551 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.551 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.551 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.551 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.551 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.551 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:22:32.551 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.551 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.551 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.551 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.551 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.552 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.552 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.552 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:22:32.552 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.557379+0000" 2024-09-22T03:22:32.552 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.552 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.552 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:22:32.552 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.536532+0000" 2024-09-22T03:22:32.552 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.552 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.552 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.552 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.552 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.553 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.553 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:22:32.553 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.553 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.553 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.553 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.553 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.553 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.553 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.553 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:22:32.553 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.536825+0000" 2024-09-22T03:22:32.553 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.554 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.554 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.554 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.554 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.554 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.554 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:22:32.554 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.554 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.554 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.554 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.554 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.554 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.554 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.554 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:22:32.555 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.537009+0000" 2024-09-22T03:22:32.555 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.555 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.555 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.555 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.555 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.555 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.555 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:22:32.555 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.555 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.555 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.555 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.555 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.556 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.556 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.556 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:22:32.556 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.556 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.556 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.556 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.556 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.556 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.556 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.556 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:22:32.556 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.556 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.556 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.557 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.557 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.557 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.557 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.557 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:22:32.557 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.537295+0000" 2024-09-22T03:22:32.557 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.557 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.557 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.557 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.557 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.557 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.557 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:22:32.558 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.558 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.558 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.558 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.558 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.558 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.558 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.558 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:22:32.558 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.537779+0000" 2024-09-22T03:22:32.558 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.558 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.558 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.558 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.559 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.559 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.559 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:22:32.559 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.559 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.559 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.559 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.559 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.559 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.559 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.559 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:22:32.559 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.555426+0000" 2024-09-22T03:22:32.559 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.559 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.560 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.560 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.560 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.560 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.560 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:22:32.560 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.560 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:22:32.560 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:22:32.560 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.560 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-22T03:22:32.560 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "client_request", 2024-09-22T03:22:32.560 INFO:teuthology.orchestra.run.smithi153.stdout: "id": 590, 2024-09-22T03:22:32.560 INFO:teuthology.orchestra.run.smithi153.stdout: "detail": { 2024-09-22T03:22:32.561 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.561 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:22:32.561 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.538351+0000" 2024-09-22T03:22:32.561 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.561 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.561 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:22:32.561 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.538361+0000" 2024-09-22T03:22:32.561 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.561 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.561 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.561 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.561 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.562 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.562 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:22:32.562 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.538861+0000" 2024-09-22T03:22:32.562 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.562 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.562 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.562 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.562 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.562 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.562 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:22:32.562 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.562 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.562 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.562 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.563 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.563 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.563 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.563 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:22:32.563 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.540036+0000" 2024-09-22T03:22:32.563 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.563 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.563 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.563 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.563 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.563 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.563 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:22:32.563 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.540833+0000" 2024-09-22T03:22:32.563 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.564 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.564 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.564 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.564 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.564 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.564 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:22:32.564 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.564 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.564 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.564 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.564 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.564 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.564 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.565 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:22:32.565 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.564847+0000" 2024-09-22T03:22:32.565 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.565 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.565 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:22:32.565 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.541112+0000" 2024-09-22T03:22:32.565 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.565 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.565 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.565 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.565 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.565 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.565 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:22:32.565 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.566 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.566 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.566 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.566 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.566 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.566 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.566 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:22:32.566 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.541666+0000" 2024-09-22T03:22:32.566 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.566 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.566 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.566 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.567 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.567 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.567 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:22:32.567 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.567 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.567 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.567 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.567 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.567 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.567 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.567 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:22:32.567 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.541834+0000" 2024-09-22T03:22:32.567 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.568 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.568 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.568 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.568 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.568 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.568 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:22:32.568 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.568 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.568 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.568 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.568 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.568 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.568 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.568 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:22:32.569 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.569 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.569 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.569 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.569 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.569 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.569 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.569 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:22:32.569 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.569 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.569 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.569 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.569 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.570 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.570 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.570 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:22:32.570 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.542035+0000" 2024-09-22T03:22:32.570 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.570 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.570 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.570 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.570 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.570 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.570 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:22:32.570 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.570 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.570 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.571 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.571 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.571 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.571 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.571 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:22:32.571 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.543106+0000" 2024-09-22T03:22:32.571 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.571 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.571 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.571 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.571 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.571 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.571 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:22:32.571 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.572 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.572 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.572 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.572 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.572 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.572 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.572 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:22:32.572 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.564014+0000" 2024-09-22T03:22:32.572 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.572 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.572 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.572 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.572 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.573 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.573 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:22:32.573 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.573 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:22:32.573 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:22:32.573 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.573 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-22T03:22:32.573 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "client_request", 2024-09-22T03:22:32.573 INFO:teuthology.orchestra.run.smithi153.stdout: "id": 592, 2024-09-22T03:22:32.573 INFO:teuthology.orchestra.run.smithi153.stdout: "detail": { 2024-09-22T03:22:32.573 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.573 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:22:32.573 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.546018+0000" 2024-09-22T03:22:32.573 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.574 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.574 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:22:32.574 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.546024+0000" 2024-09-22T03:22:32.574 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.574 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.574 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.574 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.574 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.574 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.574 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:22:32.574 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.546554+0000" 2024-09-22T03:22:32.574 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.574 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.574 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.575 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.575 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.575 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.575 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:22:32.575 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.575 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.575 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.575 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.575 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.575 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.575 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.575 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:22:32.575 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.546654+0000" 2024-09-22T03:22:32.575 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.576 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.576 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.576 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.576 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.576 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.576 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:22:32.576 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.547725+0000" 2024-09-22T03:22:32.576 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.576 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.576 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.576 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.576 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.576 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.576 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:22:32.577 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.577 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.577 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.577 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.577 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.577 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.577 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.577 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:22:32.577 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.568599+0000" 2024-09-22T03:22:32.577 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.577 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.577 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:22:32.577 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.548079+0000" 2024-09-22T03:22:32.577 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.578 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.578 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.578 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.578 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.578 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.578 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:22:32.578 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.578 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.578 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.578 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.578 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.578 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.578 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.578 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:22:32.579 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.548536+0000" 2024-09-22T03:22:32.579 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.579 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.579 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.579 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.579 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.579 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.579 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:22:32.579 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.579 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.579 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.579 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.580 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.580 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.580 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.580 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:22:32.580 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.549642+0000" 2024-09-22T03:22:32.580 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.580 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.580 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.580 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.580 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.580 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.580 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:22:32.580 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.580 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.581 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.581 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.581 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.581 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.581 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.581 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:22:32.581 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.581 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.581 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.581 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.581 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.581 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.581 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.581 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:22:32.582 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.582 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.582 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.582 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.582 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.582 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.582 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.582 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:22:32.582 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.550104+0000" 2024-09-22T03:22:32.582 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.582 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.582 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.582 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.582 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.583 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.583 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:22:32.583 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.583 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.583 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.583 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.583 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.583 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.583 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.583 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:22:32.583 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.550582+0000" 2024-09-22T03:22:32.583 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.583 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.583 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.584 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.584 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.584 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.584 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:22:32.584 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.584 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.584 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.584 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.584 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.584 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.584 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.584 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:22:32.584 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.567682+0000" 2024-09-22T03:22:32.584 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.585 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.585 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.585 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.585 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.585 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.585 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:22:32.585 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.585 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:22:32.585 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:22:32.585 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.585 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-22T03:22:32.585 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "client_request", 2024-09-22T03:22:32.585 INFO:teuthology.orchestra.run.smithi153.stdout: "id": 594, 2024-09-22T03:22:32.585 INFO:teuthology.orchestra.run.smithi153.stdout: "detail": { 2024-09-22T03:22:32.586 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.586 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:22:32.586 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.550946+0000" 2024-09-22T03:22:32.586 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.586 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.586 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:22:32.586 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.550952+0000" 2024-09-22T03:22:32.586 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.586 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.586 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.586 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.586 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.586 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.586 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:22:32.587 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.551110+0000" 2024-09-22T03:22:32.587 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.587 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.587 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.587 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.587 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.587 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.587 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:22:32.587 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.587 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.587 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.587 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.587 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.587 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.588 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.588 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:22:32.588 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.551448+0000" 2024-09-22T03:22:32.588 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.588 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.588 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.588 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.588 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.588 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.588 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:22:32.588 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.552551+0000" 2024-09-22T03:22:32.588 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.588 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.589 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.589 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.589 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.589 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.589 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:22:32.589 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.589 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.589 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.589 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.589 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.589 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.589 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.589 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:22:32.589 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.572206+0000" 2024-09-22T03:22:32.590 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.590 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.590 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:22:32.590 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.552954+0000" 2024-09-22T03:22:32.590 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.590 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.590 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.590 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.590 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.590 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.590 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:22:32.590 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.590 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.590 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.591 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.591 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.591 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.591 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.591 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:22:32.591 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.553250+0000" 2024-09-22T03:22:32.591 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.591 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.591 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.591 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.591 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.591 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.592 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:22:32.592 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.592 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.592 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.592 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.592 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.592 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.592 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.592 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:22:32.592 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.555168+0000" 2024-09-22T03:22:32.592 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.592 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.592 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.593 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.593 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.593 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.593 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:22:32.593 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.593 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.593 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.593 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.593 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.593 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.593 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.593 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:22:32.593 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.594 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.594 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.594 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.594 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.594 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.594 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.594 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:22:32.594 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.594 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.594 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.594 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.594 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.595 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.595 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.595 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:22:32.595 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.556379+0000" 2024-09-22T03:22:32.595 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.595 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.595 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.595 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.595 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.595 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.595 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:22:32.595 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.595 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.595 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.596 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.596 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.596 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.596 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.596 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:22:32.596 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.557525+0000" 2024-09-22T03:22:32.596 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.596 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.596 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.596 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.596 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.596 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.596 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:22:32.597 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.597 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.597 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.597 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.597 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.597 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.597 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.597 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:22:32.597 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.571434+0000" 2024-09-22T03:22:32.597 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.597 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.597 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.597 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.598 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.598 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.598 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:22:32.598 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.598 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:22:32.598 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:22:32.598 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.598 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-22T03:22:32.598 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "client_request", 2024-09-22T03:22:32.598 INFO:teuthology.orchestra.run.smithi153.stdout: "id": 724, 2024-09-22T03:22:32.598 INFO:teuthology.orchestra.run.smithi153.stdout: "detail": { 2024-09-22T03:22:32.598 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.598 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:22:32.598 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.228669+0000" 2024-09-22T03:22:32.599 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.599 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.599 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:22:32.599 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.228673+0000" 2024-09-22T03:22:32.599 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.599 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.599 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.599 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.599 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.599 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.599 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:22:32.599 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.228742+0000" 2024-09-22T03:22:32.600 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.600 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.600 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.600 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.600 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.600 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.600 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:22:32.600 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.600 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.600 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.600 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.600 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.600 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.600 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.601 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:22:32.601 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.229004+0000" 2024-09-22T03:22:32.601 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.601 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.601 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.601 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.601 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.601 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.601 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:22:32.601 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.229235+0000" 2024-09-22T03:22:32.601 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.601 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.601 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.602 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.602 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.602 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.602 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:22:32.602 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.602 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.602 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.602 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.602 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.602 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.602 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.602 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:22:32.602 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.241297+0000" 2024-09-22T03:22:32.603 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.603 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.603 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:22:32.603 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.229681+0000" 2024-09-22T03:22:32.603 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.603 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.603 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.603 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.603 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.603 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.603 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:22:32.603 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.603 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.604 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.604 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.604 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.604 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.604 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.604 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:22:32.604 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.229793+0000" 2024-09-22T03:22:32.604 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.604 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.604 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.604 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.604 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.604 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.605 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:22:32.605 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.605 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.605 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.605 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.605 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.605 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.605 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.605 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:22:32.605 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.229887+0000" 2024-09-22T03:22:32.605 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.605 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.606 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.606 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.606 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.606 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.606 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:22:32.606 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.606 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.606 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.606 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.606 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.606 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.606 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.606 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:22:32.607 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.607 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.607 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.607 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.607 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.607 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.607 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.607 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:22:32.607 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.607 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.607 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.607 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.607 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.607 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.608 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.608 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:22:32.608 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.230051+0000" 2024-09-22T03:22:32.608 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.608 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.608 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.608 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.608 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.608 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.608 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:22:32.608 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.608 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.608 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.608 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.609 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.609 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.609 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.609 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:22:32.609 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.232114+0000" 2024-09-22T03:22:32.609 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.609 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.609 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.609 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.609 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.609 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.609 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:22:32.609 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.610 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.610 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.610 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.610 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.610 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.610 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.610 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:22:32.610 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.240744+0000" 2024-09-22T03:22:32.610 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.610 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.610 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.610 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.610 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.610 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.610 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:22:32.611 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.611 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:22:32.611 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:22:32.611 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.611 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-22T03:22:32.611 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "client_request", 2024-09-22T03:22:32.611 INFO:teuthology.orchestra.run.smithi153.stdout: "id": 726, 2024-09-22T03:22:32.611 INFO:teuthology.orchestra.run.smithi153.stdout: "detail": { 2024-09-22T03:22:32.611 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.611 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:22:32.611 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.231001+0000" 2024-09-22T03:22:32.611 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.611 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.612 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:22:32.612 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.231007+0000" 2024-09-22T03:22:32.612 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.612 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.612 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.612 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.612 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.612 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.612 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:22:32.612 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.231095+0000" 2024-09-22T03:22:32.612 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.612 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.612 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.612 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.613 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.613 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.613 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:22:32.613 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.613 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.613 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.613 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.613 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.613 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.613 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.613 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:22:32.613 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.231145+0000" 2024-09-22T03:22:32.613 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.613 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.614 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.614 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.614 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.614 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.614 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:22:32.614 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.231200+0000" 2024-09-22T03:22:32.614 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.614 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.614 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.614 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.614 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.614 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.614 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:22:32.615 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.615 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.615 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.615 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.615 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.615 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.615 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.615 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:22:32.615 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.246025+0000" 2024-09-22T03:22:32.615 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.615 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.615 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:22:32.615 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.231413+0000" 2024-09-22T03:22:32.616 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.616 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.616 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.616 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.616 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.616 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.616 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:22:32.616 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.616 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.616 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.616 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.616 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.616 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.616 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.617 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:22:32.617 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.231513+0000" 2024-09-22T03:22:32.617 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.617 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.617 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.617 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.617 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.617 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.617 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:22:32.617 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.617 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.617 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.617 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.617 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.617 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.618 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.618 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:22:32.618 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.231681+0000" 2024-09-22T03:22:32.618 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.618 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.618 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.618 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.618 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.618 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.618 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:22:32.618 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.618 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.618 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.619 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.619 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.619 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.619 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.619 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:22:32.619 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.619 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.619 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.619 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.619 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.619 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.619 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.619 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:22:32.619 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.619 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.620 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.620 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.620 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.620 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.620 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.620 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:22:32.620 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.232005+0000" 2024-09-22T03:22:32.620 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.620 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.620 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.620 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.620 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.620 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.620 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:22:32.621 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.621 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.621 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.621 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.621 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.621 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.621 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.621 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:22:32.621 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.234094+0000" 2024-09-22T03:22:32.621 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.621 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.621 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.621 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.621 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.622 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.622 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:22:32.622 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.622 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.622 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.622 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.622 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.622 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.622 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.622 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:22:32.622 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.244810+0000" 2024-09-22T03:22:32.622 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.622 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.622 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.623 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.623 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.623 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.623 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:22:32.623 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.623 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:22:32.623 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:22:32.623 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.623 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-22T03:22:32.623 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "client_request", 2024-09-22T03:22:32.623 INFO:teuthology.orchestra.run.smithi153.stdout: "id": 712, 2024-09-22T03:22:32.623 INFO:teuthology.orchestra.run.smithi153.stdout: "detail": { 2024-09-22T03:22:32.623 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.623 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:22:32.623 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.202458+0000" 2024-09-22T03:22:32.624 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.624 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.624 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:22:32.624 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.202469+0000" 2024-09-22T03:22:32.624 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.624 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.624 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.624 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.624 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.624 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.624 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:22:32.624 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.202688+0000" 2024-09-22T03:22:32.624 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.624 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.625 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.625 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.625 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.625 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.625 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:22:32.625 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.625 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.625 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.625 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.625 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.625 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.625 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.625 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:22:32.625 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.202774+0000" 2024-09-22T03:22:32.626 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.626 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.626 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.626 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.626 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.626 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.626 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:22:32.626 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.209114+0000" 2024-09-22T03:22:32.626 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.626 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.627 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.627 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.627 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.627 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.627 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:22:32.627 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.627 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.627 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.627 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.627 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.627 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.627 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.627 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:22:32.628 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.628 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.628 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.628 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:22:32.628 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.211201+0000" 2024-09-22T03:22:32.628 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.628 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.628 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.628 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.628 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.628 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.628 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:22:32.628 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.628 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.629 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.629 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.629 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.629 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.629 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.629 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:22:32.629 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.211533+0000" 2024-09-22T03:22:32.629 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.629 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.629 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.629 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.629 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.629 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.629 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:22:32.630 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.630 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.630 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.630 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.630 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.630 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.630 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.630 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:22:32.630 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.212023+0000" 2024-09-22T03:22:32.630 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.630 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.630 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.630 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.631 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.631 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.631 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:22:32.631 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.631 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.631 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.631 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.631 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.631 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.631 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.631 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:22:32.631 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.631 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.632 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.632 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.632 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.632 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.632 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.632 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:22:32.632 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.632 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.632 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.632 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.632 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.632 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.632 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.632 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:22:32.633 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.213131+0000" 2024-09-22T03:22:32.633 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.633 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.633 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.633 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.633 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.633 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.633 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:22:32.633 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.633 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.633 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.633 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.633 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.634 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.634 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.634 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:22:32.634 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.241155+0000" 2024-09-22T03:22:32.634 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.634 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.634 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.634 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.634 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.634 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.634 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:22:32.634 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.634 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.635 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.635 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.635 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.635 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.635 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.635 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:22:32.635 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.635 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.635 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.635 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.635 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.635 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.635 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.636 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:22:32.636 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.636 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:22:32.636 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:22:32.636 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.636 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-22T03:22:32.636 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "client_request", 2024-09-22T03:22:32.636 INFO:teuthology.orchestra.run.smithi153.stdout: "id": 714, 2024-09-22T03:22:32.636 INFO:teuthology.orchestra.run.smithi153.stdout: "detail": { 2024-09-22T03:22:32.636 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.636 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:22:32.636 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.208368+0000" 2024-09-22T03:22:32.636 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.637 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.637 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:22:32.637 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.208377+0000" 2024-09-22T03:22:32.637 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.637 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.637 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.637 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.637 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.637 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.637 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:22:32.637 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.208569+0000" 2024-09-22T03:22:32.637 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.637 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.637 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.638 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.638 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.638 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.638 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:22:32.638 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.638 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.638 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.638 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.638 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.638 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.638 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.638 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:22:32.638 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.209185+0000" 2024-09-22T03:22:32.639 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.639 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.639 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.639 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.639 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.639 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.639 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:22:32.639 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.212076+0000" 2024-09-22T03:22:32.639 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.639 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.639 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.639 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.639 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.639 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.639 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:22:32.640 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.640 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.640 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.640 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.640 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.640 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.640 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.640 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:22:32.640 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.640 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.640 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.640 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:22:32.640 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.212866+0000" 2024-09-22T03:22:32.640 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.641 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.641 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.641 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.641 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.641 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.641 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:22:32.641 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.641 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.641 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.641 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.641 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.641 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.641 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.641 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:22:32.642 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.213290+0000" 2024-09-22T03:22:32.642 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.642 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.642 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.642 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.642 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.642 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.642 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:22:32.642 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.642 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.642 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.642 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.642 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.642 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.643 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.643 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:22:32.643 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.213451+0000" 2024-09-22T03:22:32.643 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.643 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.643 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.643 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.643 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.643 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.643 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:22:32.643 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.643 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.643 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.643 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.644 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.644 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.644 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.644 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:22:32.644 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.644 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.644 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.644 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.644 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.644 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.644 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.644 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:22:32.644 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.644 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.644 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.645 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.645 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.645 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.645 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.645 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:22:32.645 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.213999+0000" 2024-09-22T03:22:32.645 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.645 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.645 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.645 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.645 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.645 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.645 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:22:32.645 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.646 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.646 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.646 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.646 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.241264+0000" 2024-09-22T03:22:32.646 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.646 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.646 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:22:32.646 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.241401+0000" 2024-09-22T03:22:32.646 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.646 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.646 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.646 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.646 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.646 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.647 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:22:32.647 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.647 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.647 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.647 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.647 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.647 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.647 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.647 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:22:32.647 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.647 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.647 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.647 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.647 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.648 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.648 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.648 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:22:32.648 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.648 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:22:32.648 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:22:32.648 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.648 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-22T03:22:32.648 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "client_request", 2024-09-22T03:22:32.648 INFO:teuthology.orchestra.run.smithi153.stdout: "id": 715, 2024-09-22T03:22:32.648 INFO:teuthology.orchestra.run.smithi153.stdout: "detail": { 2024-09-22T03:22:32.648 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.648 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:22:32.648 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.211216+0000" 2024-09-22T03:22:32.648 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.649 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.649 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:22:32.649 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.211227+0000" 2024-09-22T03:22:32.649 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.649 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.649 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.649 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.649 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.649 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.649 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:22:32.649 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.212182+0000" 2024-09-22T03:22:32.649 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.650 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.650 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.650 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.650 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.650 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.650 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:22:32.650 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.650 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.650 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.650 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.650 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.650 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.650 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.650 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:22:32.651 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.212481+0000" 2024-09-22T03:22:32.651 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.651 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.651 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.651 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.651 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.651 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.651 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:22:32.651 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.214473+0000" 2024-09-22T03:22:32.651 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.651 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.651 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.651 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.651 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.652 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.652 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:22:32.652 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.652 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.652 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.652 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.652 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.652 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.652 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.652 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:22:32.652 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.652 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.652 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.653 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:22:32.653 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.214923+0000" 2024-09-22T03:22:32.653 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.653 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.653 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.653 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.653 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.653 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.653 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:22:32.653 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.653 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.653 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.653 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.653 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.654 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.654 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.654 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:22:32.654 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.215088+0000" 2024-09-22T03:22:32.654 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.654 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.654 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.654 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.654 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.654 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.654 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:22:32.654 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.654 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.654 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.655 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.655 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.655 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.655 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.655 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:22:32.655 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.215492+0000" 2024-09-22T03:22:32.655 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.655 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.655 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.655 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.655 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.655 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.655 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:22:32.656 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.656 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.656 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.656 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.656 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.656 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.656 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.656 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:22:32.656 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.656 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.656 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.656 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.656 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.656 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.657 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.657 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:22:32.657 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.657 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.657 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.657 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.657 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.657 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.657 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.657 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:22:32.657 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.216406+0000" 2024-09-22T03:22:32.657 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.657 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.657 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.658 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.658 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.658 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.658 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:22:32.658 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.658 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.658 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.658 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.658 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.241096+0000" 2024-09-22T03:22:32.658 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.658 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.658 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:22:32.658 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.246263+0000" 2024-09-22T03:22:32.658 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.659 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.659 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.659 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.659 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.659 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.659 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:22:32.659 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.659 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.659 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.659 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.659 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.659 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.659 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.659 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:22:32.660 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.660 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.660 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.660 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.660 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.660 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.660 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.660 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:22:32.660 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.660 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:22:32.660 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:22:32.660 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.660 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-22T03:22:32.661 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "client_request", 2024-09-22T03:22:32.661 INFO:teuthology.orchestra.run.smithi153.stdout: "id": 717, 2024-09-22T03:22:32.661 INFO:teuthology.orchestra.run.smithi153.stdout: "detail": { 2024-09-22T03:22:32.661 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.661 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:22:32.661 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.216961+0000" 2024-09-22T03:22:32.661 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.661 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.661 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:22:32.661 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.216972+0000" 2024-09-22T03:22:32.661 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.661 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.661 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.661 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.662 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.662 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.662 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:22:32.662 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.217099+0000" 2024-09-22T03:22:32.662 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.662 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.662 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.662 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.662 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.662 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.662 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:22:32.662 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.662 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.663 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.663 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.663 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.663 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.663 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.663 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:22:32.663 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.217198+0000" 2024-09-22T03:22:32.663 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.663 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.663 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.663 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.663 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.663 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.663 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:22:32.664 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.218981+0000" 2024-09-22T03:22:32.664 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.664 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.664 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.664 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.664 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.664 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.664 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:22:32.664 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.664 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.664 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.664 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.664 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.664 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.665 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.665 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:22:32.665 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.665 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.665 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.665 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:22:32.665 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.219445+0000" 2024-09-22T03:22:32.665 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.665 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.665 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.665 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.665 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.665 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.665 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:22:32.665 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.666 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.666 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.666 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.666 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.666 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.666 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.666 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:22:32.666 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.219644+0000" 2024-09-22T03:22:32.666 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.666 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.666 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.666 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.666 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.666 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.667 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:22:32.667 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.667 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.667 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.667 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.667 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.667 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.667 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.667 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:22:32.667 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.219826+0000" 2024-09-22T03:22:32.667 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.667 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.667 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.667 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.668 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.668 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.668 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:22:32.668 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.668 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.668 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.668 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.668 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.668 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.668 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.668 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:22:32.668 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.668 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.668 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.669 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.669 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.669 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.669 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.669 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:22:32.669 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.669 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.669 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.669 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.669 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.669 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.669 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.669 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:22:32.669 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.220116+0000" 2024-09-22T03:22:32.669 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.670 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.670 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.670 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.670 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.670 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.670 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:22:32.670 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.670 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.670 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.670 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.670 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.241011+0000" 2024-09-22T03:22:32.670 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.671 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.671 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:22:32.671 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.250103+0000" 2024-09-22T03:22:32.671 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.671 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.671 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.671 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.671 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.671 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.671 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:22:32.671 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.671 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.671 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.671 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.671 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.672 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.672 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.672 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:22:32.672 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.672 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.672 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.672 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.672 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.672 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.672 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.672 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:22:32.672 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.673 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:22:32.673 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:22:32.673 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.673 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-22T03:22:32.673 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "client_request", 2024-09-22T03:22:32.673 INFO:teuthology.orchestra.run.smithi153.stdout: "id": 719, 2024-09-22T03:22:32.673 INFO:teuthology.orchestra.run.smithi153.stdout: "detail": { 2024-09-22T03:22:32.673 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.673 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:22:32.673 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.219241+0000" 2024-09-22T03:22:32.673 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.673 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.673 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:22:32.673 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.219250+0000" 2024-09-22T03:22:32.673 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.674 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.674 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.674 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.674 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.674 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.674 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:22:32.674 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.219554+0000" 2024-09-22T03:22:32.674 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.674 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.674 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.674 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.674 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.674 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.675 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:22:32.675 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.675 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.675 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.675 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.675 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.675 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.675 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.675 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:22:32.675 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.219712+0000" 2024-09-22T03:22:32.675 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.675 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.675 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.675 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.676 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.676 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.676 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:22:32.676 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.222485+0000" 2024-09-22T03:22:32.676 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.676 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.676 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.676 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.676 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.676 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.676 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:22:32.676 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.676 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.676 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.677 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.677 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.677 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.677 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.677 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:22:32.677 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.677 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.677 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.677 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:22:32.677 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.223052+0000" 2024-09-22T03:22:32.677 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.677 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.677 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.677 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.677 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.678 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.678 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:22:32.678 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.678 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.678 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.678 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.678 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.678 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.678 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.678 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:22:32.678 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.224762+0000" 2024-09-22T03:22:32.678 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.679 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.679 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.679 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.679 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.679 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.679 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:22:32.679 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.679 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.679 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.679 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.679 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.679 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.679 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.680 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:22:32.680 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.224988+0000" 2024-09-22T03:22:32.680 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.680 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.680 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.680 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.680 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.680 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.680 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:22:32.680 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.680 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.680 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.680 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.681 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.681 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.681 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.681 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:22:32.681 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.681 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.681 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.681 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.681 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.681 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.681 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.681 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:22:32.681 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.681 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.682 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.682 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.682 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.682 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.682 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.682 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:22:32.682 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.225279+0000" 2024-09-22T03:22:32.682 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.682 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.682 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.682 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.682 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.682 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.683 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:22:32.683 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.683 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.683 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.683 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.683 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.240868+0000" 2024-09-22T03:22:32.683 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.683 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.683 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:22:32.683 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.258318+0000" 2024-09-22T03:22:32.683 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.683 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.683 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.683 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.684 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.684 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.684 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:22:32.684 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.684 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.684 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.684 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.684 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.684 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.684 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.684 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:22:32.684 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.684 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.684 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.685 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.685 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.685 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.685 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.685 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:22:32.685 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.685 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:22:32.685 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:22:32.685 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.685 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-22T03:22:32.685 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "client_request", 2024-09-22T03:22:32.685 INFO:teuthology.orchestra.run.smithi153.stdout: "id": 721, 2024-09-22T03:22:32.685 INFO:teuthology.orchestra.run.smithi153.stdout: "detail": { 2024-09-22T03:22:32.685 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.686 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:22:32.686 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.225777+0000" 2024-09-22T03:22:32.686 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.686 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.686 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:22:32.686 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.225787+0000" 2024-09-22T03:22:32.686 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.686 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.686 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.686 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.686 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.686 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.686 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:22:32.686 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.225900+0000" 2024-09-22T03:22:32.687 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.687 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.687 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.687 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.687 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.687 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.687 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:22:32.687 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.687 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.687 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.687 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.687 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.687 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.687 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.688 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:22:32.688 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.227217+0000" 2024-09-22T03:22:32.688 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.688 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.688 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.688 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.688 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.688 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.688 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:22:32.688 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.233181+0000" 2024-09-22T03:22:32.688 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.688 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.688 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.688 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.689 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.689 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.689 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:22:32.689 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.689 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.689 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.689 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.689 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.689 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.689 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.689 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:22:32.689 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.689 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.689 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.689 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:22:32.690 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.233905+0000" 2024-09-22T03:22:32.690 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.690 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.690 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.690 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.690 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.690 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.690 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:22:32.690 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.690 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.690 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.690 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.690 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.691 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.691 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.691 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:22:32.691 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.235483+0000" 2024-09-22T03:22:32.691 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.691 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.691 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.691 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.691 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.691 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.691 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:22:32.691 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.691 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.691 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.692 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.692 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.692 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.692 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.692 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:22:32.692 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.236296+0000" 2024-09-22T03:22:32.692 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.692 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.692 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.692 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.692 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.692 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.692 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:22:32.693 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.693 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.693 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.693 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.693 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.693 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.693 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.693 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:22:32.693 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.693 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.694 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.694 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.694 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.694 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.694 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.694 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:22:32.694 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.694 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.694 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.694 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.694 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.695 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.695 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.695 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:22:32.695 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.236587+0000" 2024-09-22T03:22:32.695 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.695 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.695 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.695 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.695 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.695 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.695 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:22:32.695 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.695 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.696 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.696 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.696 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.241211+0000" 2024-09-22T03:22:32.696 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.696 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.696 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:22:32.696 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.262612+0000", 2024-09-22T03:22:32.696 INFO:teuthology.orchestra.run.smithi153.stdout: "blocker": { 2024-09-22T03:22:32.696 INFO:teuthology.orchestra.run.smithi153.stdout: "op_type": "CommonPGPipeline::process", 2024-09-22T03:22:32.696 INFO:teuthology.orchestra.run.smithi153.stdout: "detail": { 2024-09-22T03:22:32.696 INFO:teuthology.orchestra.run.smithi153.stdout: "waiting": 1, 2024-09-22T03:22:32.696 INFO:teuthology.orchestra.run.smithi153.stdout: "held_by_operation_id": 719 2024-09-22T03:22:32.696 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:22:32.696 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:22:32.697 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.697 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.697 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.697 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.697 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.697 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.697 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:22:32.697 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.697 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.697 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.697 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.697 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.697 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.697 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.698 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:22:32.698 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.698 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.698 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.698 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.698 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.698 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.698 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.698 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:22:32.698 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.698 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:22:32.698 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:22:32.698 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.698 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-22T03:22:32.699 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "client_request", 2024-09-22T03:22:32.699 INFO:teuthology.orchestra.run.smithi153.stdout: "id": 723, 2024-09-22T03:22:32.699 INFO:teuthology.orchestra.run.smithi153.stdout: "detail": { 2024-09-22T03:22:32.699 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.699 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:22:32.699 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.227981+0000" 2024-09-22T03:22:32.699 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.699 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.699 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:22:32.699 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.227987+0000" 2024-09-22T03:22:32.699 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.699 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.699 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.699 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.700 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.700 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.700 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:22:32.700 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.228100+0000" 2024-09-22T03:22:32.700 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.700 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.700 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.700 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.700 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.700 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.700 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:22:32.700 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.700 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.700 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.701 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.701 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.701 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.701 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.701 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:22:32.701 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.228281+0000" 2024-09-22T03:22:32.701 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.701 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.701 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.701 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.701 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.701 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.701 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:22:32.701 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.233388+0000" 2024-09-22T03:22:32.701 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.702 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.702 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.702 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.702 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.702 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.702 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:22:32.702 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.702 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.702 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.702 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.702 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.702 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.702 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.702 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:22:32.703 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.703 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.703 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.703 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:22:32.703 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.235887+0000" 2024-09-22T03:22:32.703 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.703 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.703 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.703 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.703 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.703 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.703 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:22:32.703 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.703 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.704 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.704 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.704 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.704 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.704 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.704 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:22:32.704 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.236791+0000" 2024-09-22T03:22:32.704 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.704 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.704 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.704 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.704 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.704 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.704 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:22:32.705 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.705 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.705 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.705 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.705 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.705 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.705 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.705 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:22:32.705 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.236983+0000" 2024-09-22T03:22:32.705 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.705 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.705 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.705 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.705 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.706 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.706 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:22:32.706 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.706 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.706 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.706 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.706 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.706 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.706 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.706 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:22:32.706 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.706 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.706 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.707 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.707 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.707 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.707 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.707 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:22:32.707 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.707 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.707 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.707 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.707 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.707 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.707 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.707 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:22:32.708 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.237603+0000" 2024-09-22T03:22:32.708 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.708 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.708 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.708 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.708 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.708 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.708 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:22:32.708 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.708 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.708 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.708 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.708 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.240948+0000" 2024-09-22T03:22:32.708 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.708 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.709 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:22:32.709 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.709 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.709 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.709 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.709 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.709 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.709 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.709 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:22:32.709 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.709 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.709 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.709 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.710 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.710 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.710 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.710 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:22:32.710 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.710 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.710 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.710 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.710 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.710 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.710 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.710 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:22:32.710 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.710 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:22:32.711 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:22:32.711 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.711 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-22T03:22:32.711 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "client_request", 2024-09-22T03:22:32.711 INFO:teuthology.orchestra.run.smithi153.stdout: "id": 652, 2024-09-22T03:22:32.711 INFO:teuthology.orchestra.run.smithi153.stdout: "detail": { 2024-09-22T03:22:32.711 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.711 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:22:32.711 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.826994+0000" 2024-09-22T03:22:32.711 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.711 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.711 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:22:32.711 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.827003+0000" 2024-09-22T03:22:32.711 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.712 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.712 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.712 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.712 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.712 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.712 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:22:32.712 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.827079+0000" 2024-09-22T03:22:32.712 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.712 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.712 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.712 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.712 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.712 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.712 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:22:32.712 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.713 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.713 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.713 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.713 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.713 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.713 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.713 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:22:32.713 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.827124+0000" 2024-09-22T03:22:32.713 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.713 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.713 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.713 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.713 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.713 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.714 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:22:32.714 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.827338+0000" 2024-09-22T03:22:32.714 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.714 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.714 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.714 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.714 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.714 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.714 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:22:32.714 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.714 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.714 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.714 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.714 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.715 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.715 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.715 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:22:32.715 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.913188+0000" 2024-09-22T03:22:32.715 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.715 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.715 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:22:32.715 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.827574+0000" 2024-09-22T03:22:32.715 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.715 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.715 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.715 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.715 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.716 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.716 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:22:32.716 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.716 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.716 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.716 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.716 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.716 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.716 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.716 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:22:32.716 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.827666+0000" 2024-09-22T03:22:32.716 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.716 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.716 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.716 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.717 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.717 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.717 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:22:32.717 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.717 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.717 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.717 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.717 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.717 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.717 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.717 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:22:32.717 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.827748+0000" 2024-09-22T03:22:32.717 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.717 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.718 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.718 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.718 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.718 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.718 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:22:32.718 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.718 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.718 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.718 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.718 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.718 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.718 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.718 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:22:32.719 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.719 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.719 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.719 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.719 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.719 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.719 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.719 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:22:32.719 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.719 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.719 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.719 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.719 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.719 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.720 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.720 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:22:32.720 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.827880+0000" 2024-09-22T03:22:32.720 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.720 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.720 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.720 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.720 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.720 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.720 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:22:32.720 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.720 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.720 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.720 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.721 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.841653+0000" 2024-09-22T03:22:32.721 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.721 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.721 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:22:32.721 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.862184+0000" 2024-09-22T03:22:32.721 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.721 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.721 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.721 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.721 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.721 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.721 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:22:32.721 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.721 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.722 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.722 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.722 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.722 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.722 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.722 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:22:32.722 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.912788+0000" 2024-09-22T03:22:32.722 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.722 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.722 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.722 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.722 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.722 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.722 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:22:32.723 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.723 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:22:32.723 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:22:32.723 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.723 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-22T03:22:32.723 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "client_request", 2024-09-22T03:22:32.723 INFO:teuthology.orchestra.run.smithi153.stdout: "id": 660, 2024-09-22T03:22:32.723 INFO:teuthology.orchestra.run.smithi153.stdout: "detail": { 2024-09-22T03:22:32.723 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.723 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:22:32.723 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.840541+0000" 2024-09-22T03:22:32.723 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.723 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.723 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:22:32.723 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.840547+0000" 2024-09-22T03:22:32.724 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.724 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.724 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.724 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.724 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.724 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.724 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:22:32.724 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.840684+0000" 2024-09-22T03:22:32.724 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.724 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.724 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.724 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.724 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.724 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.725 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:22:32.725 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.725 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.725 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.725 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.725 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.725 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.725 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.725 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:22:32.725 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.840737+0000" 2024-09-22T03:22:32.725 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.725 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.725 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.725 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.725 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.726 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.726 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:22:32.726 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.840874+0000" 2024-09-22T03:22:32.726 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.726 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.726 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.726 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.726 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.726 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.726 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:22:32.726 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.726 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.726 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.726 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.727 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.727 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.727 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.727 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:22:32.727 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.917343+0000" 2024-09-22T03:22:32.727 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.727 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.727 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:22:32.727 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.841409+0000" 2024-09-22T03:22:32.727 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.727 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.727 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.727 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.727 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.728 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.728 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:22:32.728 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.728 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.728 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.728 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.728 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.728 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.728 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.728 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:22:32.728 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.842168+0000" 2024-09-22T03:22:32.728 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.728 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.729 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.729 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.729 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.729 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.729 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:22:32.729 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.729 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.729 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.729 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.729 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.729 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.729 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.729 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:22:32.729 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.842569+0000" 2024-09-22T03:22:32.730 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.730 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.730 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.730 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.730 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.730 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.730 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:22:32.730 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.730 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.730 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.730 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.730 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.730 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.730 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.731 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:22:32.731 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.731 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.731 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.731 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.731 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.731 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.731 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.731 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:22:32.731 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.731 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.731 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.731 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.731 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.731 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.732 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.732 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:22:32.732 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.844259+0000" 2024-09-22T03:22:32.732 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.732 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.732 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.732 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.732 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.732 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.732 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:22:32.732 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.732 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.732 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.732 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.733 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.845334+0000" 2024-09-22T03:22:32.733 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.733 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.733 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:22:32.733 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.865027+0000" 2024-09-22T03:22:32.733 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.733 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.733 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.733 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.733 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.733 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.733 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:22:32.733 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.733 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.734 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.734 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.734 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.734 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.734 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.734 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:22:32.734 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.917083+0000" 2024-09-22T03:22:32.734 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.734 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.734 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.734 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.734 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.734 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.734 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:22:32.734 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.735 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:22:32.735 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:22:32.735 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.735 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-22T03:22:32.735 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "client_request", 2024-09-22T03:22:32.735 INFO:teuthology.orchestra.run.smithi153.stdout: "id": 677, 2024-09-22T03:22:32.735 INFO:teuthology.orchestra.run.smithi153.stdout: "detail": { 2024-09-22T03:22:32.735 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.735 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:22:32.735 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.929083+0000" 2024-09-22T03:22:32.735 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.735 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.735 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:22:32.735 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.929093+0000" 2024-09-22T03:22:32.736 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.736 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.736 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.736 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.736 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.736 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.736 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:22:32.736 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.929386+0000" 2024-09-22T03:22:32.736 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.736 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.736 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.736 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.736 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.736 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.736 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:22:32.737 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.737 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.737 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.737 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.737 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.737 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.737 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.737 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:22:32.737 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.929479+0000" 2024-09-22T03:22:32.737 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.737 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.737 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.738 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.738 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.738 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.738 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:22:32.738 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.929696+0000" 2024-09-22T03:22:32.738 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.738 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.738 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.738 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.738 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.738 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.738 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:22:32.738 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.738 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.739 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.739 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.739 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.739 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.739 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.739 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:22:32.739 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.972441+0000" 2024-09-22T03:22:32.739 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.739 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.739 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:22:32.739 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.930066+0000" 2024-09-22T03:22:32.739 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.739 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.740 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.740 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.740 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.740 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.740 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:22:32.740 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.740 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.740 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.740 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.740 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.740 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.740 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.740 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:22:32.740 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.930242+0000" 2024-09-22T03:22:32.740 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.741 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.741 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.741 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.741 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.741 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.741 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:22:32.741 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.741 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.741 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.741 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.741 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.741 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.741 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.742 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:22:32.742 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.930415+0000" 2024-09-22T03:22:32.742 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.742 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.742 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.742 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.742 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.742 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.742 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:22:32.742 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.742 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.742 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.742 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.742 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.743 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.743 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.743 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:22:32.743 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.743 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.743 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.743 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.743 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.743 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.743 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.743 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:22:32.743 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.743 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.743 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.743 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.744 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.744 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.744 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.744 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:22:32.744 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.930665+0000" 2024-09-22T03:22:32.744 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.744 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.744 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.744 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.744 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.744 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.744 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:22:32.744 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.745 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.745 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.745 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.745 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.745 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.745 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.745 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:22:32.745 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.932565+0000" 2024-09-22T03:22:32.745 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.745 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.745 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.745 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.745 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.745 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.746 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:22:32.746 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.746 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.746 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.746 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.746 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.746 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.746 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.746 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:22:32.746 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.971540+0000" 2024-09-22T03:22:32.746 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.746 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.746 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.746 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.746 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.747 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.747 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:22:32.747 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.747 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:22:32.747 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:22:32.747 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.747 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-22T03:22:32.747 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "client_request", 2024-09-22T03:22:32.747 INFO:teuthology.orchestra.run.smithi153.stdout: "id": 678, 2024-09-22T03:22:32.747 INFO:teuthology.orchestra.run.smithi153.stdout: "detail": { 2024-09-22T03:22:32.747 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.747 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:22:32.747 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.930255+0000" 2024-09-22T03:22:32.747 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.748 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.748 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:22:32.748 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.930261+0000" 2024-09-22T03:22:32.748 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.748 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.748 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.748 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.748 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.748 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.748 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:22:32.748 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.930330+0000" 2024-09-22T03:22:32.748 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.748 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.749 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.749 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.749 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.749 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.749 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:22:32.749 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.749 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.749 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.749 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.749 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.749 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.749 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.749 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:22:32.749 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.930396+0000" 2024-09-22T03:22:32.750 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.750 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.750 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.750 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.750 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.750 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.750 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:22:32.750 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.931153+0000" 2024-09-22T03:22:32.750 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.750 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.750 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.750 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.750 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.750 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.751 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:22:32.751 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.751 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.751 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.751 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.751 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.751 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.751 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.751 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:22:32.751 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.986638+0000" 2024-09-22T03:22:32.751 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.751 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.751 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:22:32.751 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.931386+0000" 2024-09-22T03:22:32.752 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.752 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.752 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.752 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.752 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.752 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.752 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:22:32.752 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.752 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.752 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.752 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.752 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.752 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.752 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.753 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:22:32.753 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.931485+0000" 2024-09-22T03:22:32.753 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.753 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.753 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.753 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.753 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.753 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.753 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:22:32.753 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.753 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.753 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.753 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.753 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.754 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.754 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.754 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:22:32.754 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.931579+0000" 2024-09-22T03:22:32.754 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.754 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.754 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.754 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.754 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.754 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.754 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:22:32.754 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.754 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.755 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.755 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.755 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.755 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.755 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.755 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:22:32.755 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.755 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.755 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.755 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.755 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.755 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.755 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.756 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:22:32.756 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.756 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.756 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.756 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.756 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.756 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.756 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.756 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:22:32.756 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.931711+0000" 2024-09-22T03:22:32.756 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.756 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.757 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.757 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.757 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.757 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.757 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:22:32.757 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.757 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.757 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.757 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.757 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.757 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.757 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.757 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:22:32.757 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.972032+0000" 2024-09-22T03:22:32.758 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.758 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.758 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.758 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.758 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.758 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.758 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:22:32.758 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.758 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.758 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.758 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.758 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.758 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.758 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.759 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:22:32.759 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.986419+0000" 2024-09-22T03:22:32.759 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.759 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.759 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.759 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.759 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.759 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.759 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:22:32.759 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.759 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:22:32.759 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:22:32.759 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.759 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-22T03:22:32.760 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "client_request", 2024-09-22T03:22:32.760 INFO:teuthology.orchestra.run.smithi153.stdout: "id": 679, 2024-09-22T03:22:32.760 INFO:teuthology.orchestra.run.smithi153.stdout: "detail": { 2024-09-22T03:22:32.760 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.760 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:22:32.760 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.931326+0000" 2024-09-22T03:22:32.760 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.760 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.760 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:22:32.760 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.931331+0000" 2024-09-22T03:22:32.760 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.760 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.760 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.760 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.761 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.761 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.761 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:22:32.761 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.931395+0000" 2024-09-22T03:22:32.761 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.761 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.761 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.761 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.761 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.761 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.761 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:22:32.761 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.761 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.761 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.761 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.762 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.762 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.762 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.762 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:22:32.762 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.931448+0000" 2024-09-22T03:22:32.762 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.762 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.762 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.762 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.762 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.762 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.762 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:22:32.762 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.932083+0000" 2024-09-22T03:22:32.762 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.763 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.763 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.763 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.763 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.763 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.763 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:22:32.763 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.763 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.763 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.763 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.763 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.763 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.763 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.763 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:22:32.764 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.993946+0000" 2024-09-22T03:22:32.764 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.764 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.764 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:22:32.764 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.932359+0000" 2024-09-22T03:22:32.764 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.764 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.764 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.764 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.764 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.764 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.764 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:22:32.764 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.765 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.765 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.765 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.765 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.765 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.765 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.765 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:22:32.765 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.932652+0000" 2024-09-22T03:22:32.765 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.765 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.765 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.765 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.765 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.765 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.765 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:22:32.766 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.766 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.766 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.766 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.766 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.766 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.766 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.766 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:22:32.766 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.932742+0000" 2024-09-22T03:22:32.766 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.766 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.766 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.766 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.767 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.767 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.767 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:22:32.767 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.767 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.767 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.767 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.767 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.767 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.767 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.767 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:22:32.767 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.767 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.768 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.768 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.768 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.768 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.768 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.768 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:22:32.768 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.768 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.768 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.768 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.768 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.768 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.768 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.769 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:22:32.769 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.933157+0000" 2024-09-22T03:22:32.769 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.769 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.769 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.769 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.769 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.769 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.769 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:22:32.769 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.769 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.769 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.769 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.769 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.971949+0000" 2024-09-22T03:22:32.770 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.770 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.770 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:22:32.770 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.972544+0000" 2024-09-22T03:22:32.770 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.770 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.770 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.770 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.770 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.770 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.770 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:22:32.770 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.770 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.771 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.771 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.771 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.771 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.771 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.771 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:22:32.771 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.993559+0000" 2024-09-22T03:22:32.771 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.771 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.771 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.771 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.771 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.771 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.771 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:22:32.772 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.772 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:22:32.772 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:22:32.772 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.772 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-22T03:22:32.772 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "client_request", 2024-09-22T03:22:32.772 INFO:teuthology.orchestra.run.smithi153.stdout: "id": 680, 2024-09-22T03:22:32.772 INFO:teuthology.orchestra.run.smithi153.stdout: "detail": { 2024-09-22T03:22:32.772 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.772 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:22:32.772 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.933773+0000" 2024-09-22T03:22:32.772 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.772 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.772 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:22:32.773 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.933783+0000" 2024-09-22T03:22:32.773 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.773 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.773 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.773 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.773 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.773 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.773 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:22:32.773 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.933915+0000" 2024-09-22T03:22:32.773 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.773 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.773 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.773 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.774 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.774 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.774 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:22:32.774 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.774 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.774 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.774 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.774 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.774 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.774 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.774 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:22:32.774 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.934045+0000" 2024-09-22T03:22:32.774 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.775 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.775 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.775 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.775 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.775 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.775 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:22:32.775 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.934874+0000" 2024-09-22T03:22:32.775 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.775 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.775 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.775 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.775 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.775 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.776 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:22:32.776 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.776 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.776 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.776 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.776 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.776 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.776 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.776 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:22:32.776 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.000005+0000" 2024-09-22T03:22:32.776 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.776 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.776 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:22:32.777 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.935270+0000" 2024-09-22T03:22:32.777 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.777 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.777 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.777 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.777 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.777 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.777 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:22:32.777 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.777 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.777 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.777 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.777 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.777 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.778 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.778 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:22:32.778 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.935443+0000" 2024-09-22T03:22:32.778 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.778 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.778 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.778 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.778 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.778 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.778 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:22:32.778 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.778 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.779 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.779 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.779 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.779 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.779 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.779 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:22:32.779 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.935611+0000" 2024-09-22T03:22:32.779 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.779 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.779 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.779 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.779 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.779 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.779 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:22:32.780 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.780 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.780 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.780 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.780 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.780 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.780 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.780 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:22:32.780 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.780 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.780 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.780 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.780 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.780 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.781 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.781 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:22:32.781 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.781 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.781 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.781 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.781 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.781 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.781 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.781 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:22:32.781 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.935847+0000" 2024-09-22T03:22:32.781 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.781 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.781 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.781 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.782 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.782 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.782 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:22:32.782 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.782 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.782 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.782 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.782 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.972364+0000" 2024-09-22T03:22:32.782 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.782 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.782 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:22:32.782 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.974059+0000" 2024-09-22T03:22:32.782 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.783 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.783 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.783 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.783 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.783 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.783 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:22:32.783 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.783 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.783 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.783 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.783 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.783 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.783 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.783 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:22:32.784 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.999604+0000" 2024-09-22T03:22:32.784 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.784 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.784 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.784 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.784 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.784 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.784 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:22:32.784 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.784 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:22:32.784 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:22:32.784 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.784 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-22T03:22:32.784 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "client_request", 2024-09-22T03:22:32.784 INFO:teuthology.orchestra.run.smithi153.stdout: "id": 681, 2024-09-22T03:22:32.785 INFO:teuthology.orchestra.run.smithi153.stdout: "detail": { 2024-09-22T03:22:32.785 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.785 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:22:32.785 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.935955+0000" 2024-09-22T03:22:32.785 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.785 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.785 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:22:32.785 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.935961+0000" 2024-09-22T03:22:32.785 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.785 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.785 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.785 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.785 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.785 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.786 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:22:32.786 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.936040+0000" 2024-09-22T03:22:32.786 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.786 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.786 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.786 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.786 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.786 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.786 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:22:32.786 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.786 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.786 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.786 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.786 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.787 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.787 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.787 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:22:32.787 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.936096+0000" 2024-09-22T03:22:32.787 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.787 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.787 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.787 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.787 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.787 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.787 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:22:32.787 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.936260+0000" 2024-09-22T03:22:32.787 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.787 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.788 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.788 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.788 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.788 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.788 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:22:32.788 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.788 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.788 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.788 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.788 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.788 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.788 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.788 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:22:32.788 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.013295+0000" 2024-09-22T03:22:32.788 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.789 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.789 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:22:32.789 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.936494+0000" 2024-09-22T03:22:32.789 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.789 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.789 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.789 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.789 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.789 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.789 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:22:32.789 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.789 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.789 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.789 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.790 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.790 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.790 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.790 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:22:32.790 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.936594+0000" 2024-09-22T03:22:32.790 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.790 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.790 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.790 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.790 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.790 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.790 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:22:32.790 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.790 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.791 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.791 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.791 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.791 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.791 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.791 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:22:32.791 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.936720+0000" 2024-09-22T03:22:32.791 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.791 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.791 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.791 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.791 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.791 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.791 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:22:32.792 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.792 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.792 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.792 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.792 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.792 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.792 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.792 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:22:32.792 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.792 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.792 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.792 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.792 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.792 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.792 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.793 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:22:32.793 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.793 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.793 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.793 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.793 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.793 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.793 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.793 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:22:32.793 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.936864+0000" 2024-09-22T03:22:32.793 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.793 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.794 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.794 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.794 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.794 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.794 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:22:32.794 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.794 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.794 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.794 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.794 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.972110+0000" 2024-09-22T03:22:32.794 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.794 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.794 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:22:32.795 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.975805+0000" 2024-09-22T03:22:32.795 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.795 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.795 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.795 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.795 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.795 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.795 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:22:32.795 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.795 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.795 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.795 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.795 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.795 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.796 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.796 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:22:32.796 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.012997+0000" 2024-09-22T03:22:32.796 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.796 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.796 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.796 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.796 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.796 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.796 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:22:32.796 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.796 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:22:32.796 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:22:32.796 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.796 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-22T03:22:32.797 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "client_request", 2024-09-22T03:22:32.797 INFO:teuthology.orchestra.run.smithi153.stdout: "id": 682, 2024-09-22T03:22:32.797 INFO:teuthology.orchestra.run.smithi153.stdout: "detail": { 2024-09-22T03:22:32.797 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.797 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:22:32.797 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.938433+0000" 2024-09-22T03:22:32.797 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.797 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.797 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:22:32.797 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.938442+0000" 2024-09-22T03:22:32.797 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.797 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.797 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.797 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.798 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.798 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.798 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:22:32.798 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.938538+0000" 2024-09-22T03:22:32.798 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.798 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.798 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.798 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.798 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.798 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.798 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:22:32.798 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.798 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.799 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.799 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.799 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.799 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.799 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.799 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:22:32.799 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.938619+0000" 2024-09-22T03:22:32.799 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.799 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.799 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.799 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.799 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.799 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.799 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:22:32.799 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.938806+0000" 2024-09-22T03:22:32.800 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.800 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.800 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.800 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.800 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.800 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.800 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:22:32.800 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.800 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.800 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.800 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.800 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.801 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.801 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.801 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:22:32.801 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.034268+0000" 2024-09-22T03:22:32.801 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.801 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.801 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:22:32.801 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.939063+0000" 2024-09-22T03:22:32.801 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.801 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.801 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.801 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.801 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.801 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.801 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:22:32.802 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.802 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.802 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.802 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.802 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.802 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.802 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.802 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:22:32.802 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.939168+0000" 2024-09-22T03:22:32.802 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.802 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.802 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.802 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.803 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.803 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.803 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:22:32.803 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.803 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.803 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.803 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.803 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.803 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.803 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.803 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:22:32.803 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.939275+0000" 2024-09-22T03:22:32.803 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.803 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.804 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.804 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.804 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.804 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.804 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:22:32.804 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.804 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.804 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.804 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.804 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.804 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.804 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.804 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:22:32.804 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.804 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.805 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.805 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.805 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.805 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.805 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.805 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:22:32.805 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.805 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.805 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.805 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.805 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.805 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.805 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.805 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:22:32.806 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.939407+0000" 2024-09-22T03:22:32.806 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.806 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.806 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.806 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.806 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.806 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.806 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:22:32.806 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.806 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.806 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.806 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.806 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.971868+0000" 2024-09-22T03:22:32.806 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.807 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.807 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:22:32.807 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.977474+0000" 2024-09-22T03:22:32.807 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.807 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.807 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.807 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.807 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.807 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.807 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:22:32.807 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.807 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.807 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.807 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.808 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.808 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.808 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.808 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:22:32.808 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.033870+0000" 2024-09-22T03:22:32.808 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.808 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.808 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.808 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.808 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.808 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.808 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:22:32.808 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.809 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:22:32.809 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:22:32.809 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.809 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-22T03:22:32.809 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "client_request", 2024-09-22T03:22:32.809 INFO:teuthology.orchestra.run.smithi153.stdout: "id": 683, 2024-09-22T03:22:32.809 INFO:teuthology.orchestra.run.smithi153.stdout: "detail": { 2024-09-22T03:22:32.809 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.809 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:22:32.809 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.943147+0000" 2024-09-22T03:22:32.809 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.809 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.809 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:22:32.810 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.943153+0000" 2024-09-22T03:22:32.810 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.810 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.810 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.810 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.810 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.810 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.810 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:22:32.810 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.943256+0000" 2024-09-22T03:22:32.810 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.810 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.810 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.810 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.810 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.811 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.811 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:22:32.811 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.811 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.811 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.811 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.811 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.811 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.811 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.811 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:22:32.811 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.943319+0000" 2024-09-22T03:22:32.811 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.811 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.811 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.812 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.812 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.812 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.812 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:22:32.812 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.943464+0000" 2024-09-22T03:22:32.812 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.812 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.812 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.812 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.812 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.812 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.812 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:22:32.812 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.812 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.813 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.813 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.813 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.813 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.813 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.813 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:22:32.813 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.038305+0000" 2024-09-22T03:22:32.813 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.813 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.813 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:22:32.813 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.943720+0000" 2024-09-22T03:22:32.813 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.813 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.813 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.814 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.814 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.814 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.814 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:22:32.814 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.814 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.814 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.814 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.814 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.814 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.814 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.814 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:22:32.814 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.943824+0000" 2024-09-22T03:22:32.814 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.815 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.815 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.815 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.815 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.815 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.815 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:22:32.815 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.815 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.815 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.815 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.815 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.815 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.815 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.815 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:22:32.816 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.943933+0000" 2024-09-22T03:22:32.816 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.816 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.816 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.816 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.816 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.816 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.816 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:22:32.816 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.816 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.816 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.816 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.816 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.816 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.817 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.817 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:22:32.817 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.817 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.817 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.817 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.817 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.817 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.817 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.817 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:22:32.817 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.817 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.817 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.817 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.818 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.818 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.818 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.818 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:22:32.818 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.944095+0000" 2024-09-22T03:22:32.818 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.818 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.818 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.818 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.818 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.818 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.818 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:22:32.818 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.818 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.818 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.819 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.819 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.972305+0000" 2024-09-22T03:22:32.819 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.819 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.819 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:22:32.819 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.979676+0000" 2024-09-22T03:22:32.819 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.819 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.819 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.819 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.819 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.819 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.819 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:22:32.819 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.820 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.820 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.820 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.820 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.820 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.820 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.820 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:22:32.820 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.037934+0000" 2024-09-22T03:22:32.820 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.820 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.820 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.820 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.820 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.821 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.821 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:22:32.821 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.821 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:22:32.821 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:22:32.821 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.821 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-22T03:22:32.821 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "client_request", 2024-09-22T03:22:32.821 INFO:teuthology.orchestra.run.smithi153.stdout: "id": 684, 2024-09-22T03:22:32.821 INFO:teuthology.orchestra.run.smithi153.stdout: "detail": { 2024-09-22T03:22:32.821 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.821 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:22:32.821 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.946367+0000" 2024-09-22T03:22:32.821 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.822 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.822 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:22:32.822 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.946376+0000" 2024-09-22T03:22:32.822 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.822 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.822 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.822 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.822 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.822 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.822 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:22:32.822 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.946503+0000" 2024-09-22T03:22:32.822 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.822 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.823 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.823 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.823 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.823 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.823 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:22:32.823 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.823 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.823 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.823 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.823 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.823 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.824 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.824 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:22:32.824 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.946599+0000" 2024-09-22T03:22:32.824 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.824 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.824 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.824 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.824 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.824 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.824 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:22:32.824 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.946777+0000" 2024-09-22T03:22:32.824 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.824 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.824 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.825 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.825 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.825 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.825 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:22:32.825 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.825 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.825 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.825 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.825 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.825 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.825 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.825 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:22:32.825 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.052981+0000" 2024-09-22T03:22:32.825 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.826 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.826 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:22:32.826 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.947159+0000" 2024-09-22T03:22:32.826 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.826 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.826 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.826 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.826 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.826 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.826 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:22:32.826 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.826 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.826 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.826 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.827 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.827 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.827 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.827 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:22:32.827 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.947328+0000" 2024-09-22T03:22:32.827 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.827 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.827 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.827 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.827 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.827 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.827 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:22:32.827 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.827 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.828 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.828 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.828 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.828 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.828 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.828 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:22:32.828 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.947497+0000" 2024-09-22T03:22:32.828 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.828 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.828 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.828 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.828 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.828 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.828 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:22:32.829 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.829 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.829 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.829 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.829 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.829 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.829 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.829 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:22:32.829 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.829 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.829 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.829 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.829 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.829 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.830 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.830 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:22:32.830 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.830 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.830 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.830 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.830 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.830 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.830 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.830 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:22:32.830 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.947742+0000" 2024-09-22T03:22:32.830 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.830 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.830 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.831 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.831 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.831 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.831 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:22:32.831 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.831 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.831 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.831 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.831 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.972244+0000" 2024-09-22T03:22:32.831 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.831 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.831 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:22:32.831 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.981355+0000" 2024-09-22T03:22:32.831 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.832 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.832 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.832 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.832 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.832 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.832 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:22:32.832 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.832 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.832 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.832 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.832 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.832 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.832 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.832 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:22:32.833 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.052589+0000" 2024-09-22T03:22:32.833 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.833 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.833 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.833 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.833 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.833 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.833 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:22:32.833 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.833 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:22:32.833 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:22:32.833 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.833 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-22T03:22:32.833 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "client_request", 2024-09-22T03:22:32.834 INFO:teuthology.orchestra.run.smithi153.stdout: "id": 700, 2024-09-22T03:22:32.834 INFO:teuthology.orchestra.run.smithi153.stdout: "detail": { 2024-09-22T03:22:32.834 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.834 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:22:32.834 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.120593+0000" 2024-09-22T03:22:32.834 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.834 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.834 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:22:32.834 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.120607+0000" 2024-09-22T03:22:32.834 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.834 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.834 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.834 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.835 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.835 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.835 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:22:32.835 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.120741+0000" 2024-09-22T03:22:32.835 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.835 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.835 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.835 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.835 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.835 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.835 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:22:32.835 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.835 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.835 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.835 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.836 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.836 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.836 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.837 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:22:32.837 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.120836+0000" 2024-09-22T03:22:32.837 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.837 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.837 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.838 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.838 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.838 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.838 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:22:32.838 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.121057+0000" 2024-09-22T03:22:32.838 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.838 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.838 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.838 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.838 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.838 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.838 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:22:32.838 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.838 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.839 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.839 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.839 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.839 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.839 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.839 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:22:32.839 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.142648+0000" 2024-09-22T03:22:32.839 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.839 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.839 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:22:32.839 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.121349+0000" 2024-09-22T03:22:32.839 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.840 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.840 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.840 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.840 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.840 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.840 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:22:32.840 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.840 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.840 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.840 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.840 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.840 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.840 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.840 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:22:32.841 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.121457+0000" 2024-09-22T03:22:32.841 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.841 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.841 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.841 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.841 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.841 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.841 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:22:32.841 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.841 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.841 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.841 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.841 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.841 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.841 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.842 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:22:32.842 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.121563+0000" 2024-09-22T03:22:32.842 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.842 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.842 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.842 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.842 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.842 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.842 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:22:32.842 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.842 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.842 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.842 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.843 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.843 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.843 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.843 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:22:32.843 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.843 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.843 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.843 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.843 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.843 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.843 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.843 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:22:32.843 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.843 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.844 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.844 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.844 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.844 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.844 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.844 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:22:32.844 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.121755+0000" 2024-09-22T03:22:32.844 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.844 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.844 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.844 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.844 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.844 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.845 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:22:32.845 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.845 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.845 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.845 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.845 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.845 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.845 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.845 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:22:32.845 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.123348+0000" 2024-09-22T03:22:32.845 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.845 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.845 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.846 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.846 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.846 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.846 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:22:32.846 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.846 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.846 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.846 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.846 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.846 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.846 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.846 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:22:32.846 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.142267+0000" 2024-09-22T03:22:32.846 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.847 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.847 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.847 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.847 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.847 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.847 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:22:32.847 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.847 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:22:32.847 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:22:32.847 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.847 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-22T03:22:32.847 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "client_request", 2024-09-22T03:22:32.847 INFO:teuthology.orchestra.run.smithi153.stdout: "id": 702, 2024-09-22T03:22:32.847 INFO:teuthology.orchestra.run.smithi153.stdout: "detail": { 2024-09-22T03:22:32.848 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.848 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:22:32.848 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.122923+0000" 2024-09-22T03:22:32.848 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.848 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.848 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:22:32.848 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.122934+0000" 2024-09-22T03:22:32.848 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.848 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.848 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.848 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.848 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.848 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.848 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:22:32.849 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.123066+0000" 2024-09-22T03:22:32.849 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.849 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.849 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.849 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.849 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.849 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.849 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:22:32.849 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.849 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.849 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.849 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.849 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.849 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.850 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.850 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:22:32.850 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.123174+0000" 2024-09-22T03:22:32.850 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.850 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.850 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.850 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.850 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.850 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.850 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:22:32.850 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.125168+0000" 2024-09-22T03:22:32.850 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.851 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.851 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.851 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.851 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.851 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.851 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:22:32.851 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.851 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.851 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.851 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.851 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.851 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.851 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.851 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:22:32.851 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.151838+0000" 2024-09-22T03:22:32.852 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.852 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.852 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:22:32.852 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.125589+0000" 2024-09-22T03:22:32.852 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.852 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.852 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.852 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.852 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.852 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.852 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:22:32.852 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.852 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.853 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.853 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.853 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.853 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.853 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.853 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:22:32.853 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.125781+0000" 2024-09-22T03:22:32.853 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.853 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.853 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.853 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.853 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.853 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.853 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:22:32.853 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.854 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.854 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.854 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.854 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.854 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.854 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.854 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:22:32.854 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.125982+0000" 2024-09-22T03:22:32.854 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.854 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.854 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.854 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.854 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.854 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.855 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:22:32.855 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.855 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.855 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.855 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.855 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.855 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.855 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.855 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:22:32.855 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.855 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.855 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.855 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.855 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.856 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.856 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.856 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:22:32.856 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.856 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.856 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.856 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.856 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.856 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.856 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.856 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:22:32.857 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.126912+0000" 2024-09-22T03:22:32.857 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.857 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.857 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.857 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.857 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.857 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.857 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:22:32.857 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.857 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.857 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.857 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.858 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.858 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.858 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.858 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:22:32.858 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.127544+0000" 2024-09-22T03:22:32.858 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.858 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.858 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.858 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.858 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.858 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.858 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:22:32.859 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.859 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.859 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.859 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.859 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.859 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.859 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.859 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:22:32.859 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.151582+0000" 2024-09-22T03:22:32.859 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.859 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.859 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.859 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.860 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.860 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.860 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:22:32.860 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.860 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:22:32.860 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:22:32.860 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.860 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-22T03:22:32.860 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "client_request", 2024-09-22T03:22:32.860 INFO:teuthology.orchestra.run.smithi153.stdout: "id": 703, 2024-09-22T03:22:32.860 INFO:teuthology.orchestra.run.smithi153.stdout: "detail": { 2024-09-22T03:22:32.861 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.861 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:22:32.861 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.124460+0000" 2024-09-22T03:22:32.861 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.861 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.861 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:22:32.861 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.124470+0000" 2024-09-22T03:22:32.861 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.861 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.861 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.861 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.861 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.862 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.862 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:22:32.862 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.124595+0000" 2024-09-22T03:22:32.862 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.862 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.862 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.862 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.862 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.862 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.862 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:22:32.862 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.862 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.863 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.863 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.863 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.863 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.863 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.863 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:22:32.863 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.124680+0000" 2024-09-22T03:22:32.863 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.863 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.863 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.863 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.864 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.864 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.864 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:22:32.864 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.125239+0000" 2024-09-22T03:22:32.864 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.864 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.864 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.864 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.864 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.864 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.864 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:22:32.864 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.864 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.865 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.865 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.865 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.865 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.865 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.865 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:22:32.865 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.172839+0000" 2024-09-22T03:22:32.865 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.865 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.865 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:22:32.865 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.126290+0000" 2024-09-22T03:22:32.865 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.866 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.866 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.866 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.866 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.866 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.866 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:22:32.866 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.866 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.866 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.866 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.866 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.866 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.867 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.867 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:22:32.867 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.126618+0000" 2024-09-22T03:22:32.867 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.867 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.867 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.867 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.867 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.867 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.867 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:22:32.867 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.867 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.867 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.868 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.868 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.868 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.868 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.868 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:22:32.868 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.127067+0000" 2024-09-22T03:22:32.868 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.868 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.868 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.868 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.868 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.868 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.868 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:22:32.869 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.869 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.869 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.869 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.869 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.869 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.869 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.869 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:22:32.869 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.869 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.869 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.869 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.869 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.870 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.870 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.870 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:22:32.870 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.870 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.870 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.870 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.870 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.870 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.870 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.870 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:22:32.870 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.127798+0000" 2024-09-22T03:22:32.871 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.871 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.871 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.871 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.871 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.871 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.871 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:22:32.871 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.871 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.871 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.871 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.872 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.872 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.872 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.872 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:22:32.872 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.128708+0000" 2024-09-22T03:22:32.872 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.872 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.872 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.872 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.872 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.872 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.872 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:22:32.873 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.873 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.873 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.873 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.873 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.873 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.873 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.873 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:22:32.873 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.172591+0000" 2024-09-22T03:22:32.873 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.873 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.873 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.873 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.873 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.874 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.874 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:22:32.874 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.874 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:22:32.874 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:22:32.874 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.874 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-22T03:22:32.874 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "client_request", 2024-09-22T03:22:32.874 INFO:teuthology.orchestra.run.smithi153.stdout: "id": 704, 2024-09-22T03:22:32.874 INFO:teuthology.orchestra.run.smithi153.stdout: "detail": { 2024-09-22T03:22:32.874 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.874 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:22:32.875 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.128900+0000" 2024-09-22T03:22:32.875 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.875 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.875 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:22:32.875 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.128911+0000" 2024-09-22T03:22:32.875 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.875 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.875 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.875 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.875 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.875 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.875 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:22:32.875 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.129050+0000" 2024-09-22T03:22:32.876 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.876 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.876 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.876 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.876 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.876 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.876 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:22:32.876 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.876 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.876 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.876 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.876 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.876 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.877 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.877 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:22:32.877 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.129150+0000" 2024-09-22T03:22:32.877 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.877 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.877 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.877 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.877 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.877 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.877 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:22:32.877 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.130583+0000" 2024-09-22T03:22:32.877 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.877 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.877 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.878 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.878 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.878 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.878 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:22:32.878 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.878 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.878 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.878 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.878 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.878 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.879 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.879 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:22:32.879 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.173319+0000" 2024-09-22T03:22:32.879 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.879 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.879 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:22:32.879 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.130968+0000" 2024-09-22T03:22:32.879 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.879 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.879 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.879 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.879 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.879 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.880 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:22:32.880 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.880 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.880 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.880 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.880 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.880 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.880 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.880 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:22:32.880 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.131201+0000" 2024-09-22T03:22:32.880 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.880 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.880 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.881 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.881 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.881 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.881 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:22:32.881 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.881 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.881 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.881 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.881 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.881 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.881 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.881 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:22:32.882 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.131706+0000" 2024-09-22T03:22:32.882 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.882 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.882 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.882 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.882 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.882 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.882 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:22:32.882 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.882 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.883 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.883 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.883 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.883 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.883 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.883 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:22:32.883 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.883 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.883 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.883 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.883 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.883 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.884 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.884 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:22:32.884 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.884 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.884 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.884 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.884 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.884 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.884 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.884 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:22:32.884 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.132129+0000" 2024-09-22T03:22:32.884 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.885 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.885 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.885 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.885 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.885 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.885 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:22:32.885 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.885 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.885 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.885 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.885 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.132770+0000" 2024-09-22T03:22:32.885 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.885 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.886 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:22:32.886 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.133504+0000" 2024-09-22T03:22:32.886 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.886 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.886 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.886 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.886 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.886 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.886 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:22:32.886 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.886 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.887 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.887 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.887 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.887 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.887 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.887 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:22:32.887 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.173117+0000" 2024-09-22T03:22:32.887 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.887 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.887 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.887 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.887 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.887 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.888 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:22:32.888 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.888 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:22:32.888 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:22:32.888 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.888 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-22T03:22:32.888 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "client_request", 2024-09-22T03:22:32.888 INFO:teuthology.orchestra.run.smithi153.stdout: "id": 705, 2024-09-22T03:22:32.888 INFO:teuthology.orchestra.run.smithi153.stdout: "detail": { 2024-09-22T03:22:32.888 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.888 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:22:32.888 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.129362+0000" 2024-09-22T03:22:32.889 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.889 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.889 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:22:32.889 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.129366+0000" 2024-09-22T03:22:32.889 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.889 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.889 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.889 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.889 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.889 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.889 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:22:32.889 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.129441+0000" 2024-09-22T03:22:32.889 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.890 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.890 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.890 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.890 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.890 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.890 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:22:32.890 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.890 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.890 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.890 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.890 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.890 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.891 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.891 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:22:32.891 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.129481+0000" 2024-09-22T03:22:32.891 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.891 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.891 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.891 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.891 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.891 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.891 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:22:32.891 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.131025+0000" 2024-09-22T03:22:32.891 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.891 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.892 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.892 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.892 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.892 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.892 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:22:32.892 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.892 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.892 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.892 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.892 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.892 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.893 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.893 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:22:32.893 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.173751+0000" 2024-09-22T03:22:32.894 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.894 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.894 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:22:32.895 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.131536+0000" 2024-09-22T03:22:32.895 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.895 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.895 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.896 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.896 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.896 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.896 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:22:32.896 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.897 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.897 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.897 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.897 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.898 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.898 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.898 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:22:32.898 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.132047+0000" 2024-09-22T03:22:32.899 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.899 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.899 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.899 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.900 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.900 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.900 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:22:32.900 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.901 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.901 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.901 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.902 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.902 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.902 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.902 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:22:32.903 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.132378+0000" 2024-09-22T03:22:32.903 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.904 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.904 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.904 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.904 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.905 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.905 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:22:32.905 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.905 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.905 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.905 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.905 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.905 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.905 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.905 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:22:32.906 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.906 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.906 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.906 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.906 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.906 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.906 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.906 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:22:32.906 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.906 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.906 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.907 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.907 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.907 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.907 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.907 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:22:32.907 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.133013+0000" 2024-09-22T03:22:32.907 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.907 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.907 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.907 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.907 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.908 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.908 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:22:32.908 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.908 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.908 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.908 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.908 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.133278+0000" 2024-09-22T03:22:32.908 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.908 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.908 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:22:32.908 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.137333+0000" 2024-09-22T03:22:32.908 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.909 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.909 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.909 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.909 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.909 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.909 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:22:32.909 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.909 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.909 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.909 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.909 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.910 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.910 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.910 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:22:32.910 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.173560+0000" 2024-09-22T03:22:32.910 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.910 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.910 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.910 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.910 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.910 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.910 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:22:32.910 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.911 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:22:32.911 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:22:32.911 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.911 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-22T03:22:32.911 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "client_request", 2024-09-22T03:22:32.911 INFO:teuthology.orchestra.run.smithi153.stdout: "id": 706, 2024-09-22T03:22:32.911 INFO:teuthology.orchestra.run.smithi153.stdout: "detail": { 2024-09-22T03:22:32.911 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.911 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:22:32.911 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.132661+0000" 2024-09-22T03:22:32.911 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.911 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.912 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:22:32.912 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.132669+0000" 2024-09-22T03:22:32.912 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.912 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.912 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.912 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.912 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.912 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.912 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:22:32.912 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.132759+0000" 2024-09-22T03:22:32.912 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.912 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.913 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.913 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.913 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.913 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.913 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:22:32.913 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.913 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.913 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.913 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.913 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.913 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.914 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.914 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:22:32.914 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.132809+0000" 2024-09-22T03:22:32.914 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.914 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.914 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.914 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.914 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.914 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.914 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:22:32.914 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.133460+0000" 2024-09-22T03:22:32.914 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.915 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.915 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.915 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.915 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.915 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.915 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:22:32.915 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.915 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.915 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.915 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.915 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.915 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.916 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.916 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:22:32.916 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.174186+0000" 2024-09-22T03:22:32.916 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.916 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.916 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:22:32.916 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.134194+0000" 2024-09-22T03:22:32.916 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.916 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.916 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.916 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.916 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.917 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.917 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:22:32.917 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.917 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.917 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.917 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.917 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.917 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.917 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.917 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:22:32.917 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.134743+0000" 2024-09-22T03:22:32.917 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.918 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.918 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.918 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.918 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.918 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.918 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:22:32.918 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.918 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.918 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.918 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.918 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.919 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.919 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.919 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:22:32.919 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.136360+0000" 2024-09-22T03:22:32.919 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.919 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.919 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.919 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.919 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.919 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.919 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:22:32.919 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.919 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.920 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.920 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.920 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.920 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.920 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.920 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:22:32.920 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.920 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.920 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.920 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.920 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.920 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.921 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.921 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:22:32.921 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.921 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.921 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.921 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.921 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.921 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.921 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.921 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:22:32.921 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.136633+0000" 2024-09-22T03:22:32.922 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.922 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.922 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.922 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.922 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.922 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.922 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:22:32.922 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.922 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.922 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.922 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.922 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.137095+0000" 2024-09-22T03:22:32.923 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.923 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.923 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:22:32.923 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.140186+0000" 2024-09-22T03:22:32.923 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.923 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.923 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.923 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.923 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.923 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.923 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:22:32.924 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.924 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.924 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.924 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.924 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.924 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.924 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.924 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:22:32.924 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.173993+0000" 2024-09-22T03:22:32.924 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.924 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.925 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.925 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.925 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.925 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.925 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:22:32.925 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.925 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:22:32.925 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:22:32.925 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.925 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-22T03:22:32.926 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "client_request", 2024-09-22T03:22:32.926 INFO:teuthology.orchestra.run.smithi153.stdout: "id": 707, 2024-09-22T03:22:32.926 INFO:teuthology.orchestra.run.smithi153.stdout: "detail": { 2024-09-22T03:22:32.926 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.926 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:22:32.926 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.137642+0000" 2024-09-22T03:22:32.926 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.926 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.926 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:22:32.926 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.137650+0000" 2024-09-22T03:22:32.926 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.927 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.927 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.927 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.927 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.927 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.927 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:22:32.927 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.137745+0000" 2024-09-22T03:22:32.927 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.927 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.927 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.928 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.928 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.928 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.928 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:22:32.928 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.928 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.928 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.928 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.928 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.928 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.929 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.929 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:22:32.929 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.137799+0000" 2024-09-22T03:22:32.929 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.929 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.929 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.929 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.929 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.929 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.930 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:22:32.930 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.139031+0000" 2024-09-22T03:22:32.930 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.930 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.930 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.930 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.930 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.930 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.930 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:22:32.930 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.931 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.931 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.931 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.931 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.931 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.931 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.931 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:22:32.931 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.174734+0000" 2024-09-22T03:22:32.931 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.931 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.932 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:22:32.932 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.139315+0000" 2024-09-22T03:22:32.932 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.932 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.932 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.932 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.932 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.932 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.932 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:22:32.932 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.932 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.933 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.933 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.933 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.933 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.933 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.933 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:22:32.933 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.139423+0000" 2024-09-22T03:22:32.933 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.933 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.933 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.933 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.934 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.934 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.934 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:22:32.934 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.934 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.934 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.934 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.934 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.934 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.934 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.934 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:22:32.935 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.139527+0000" 2024-09-22T03:22:32.935 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.935 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.935 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.935 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.935 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.935 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.935 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:22:32.935 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.935 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.935 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.936 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.936 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.936 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.936 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.936 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:22:32.936 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.936 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.936 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.936 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.936 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.936 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.936 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.937 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:22:32.937 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.937 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.937 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.937 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.937 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.937 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.937 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.937 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:22:32.937 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.139672+0000" 2024-09-22T03:22:32.938 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.938 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.938 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.938 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.938 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.938 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.938 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:22:32.938 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.938 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.938 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.939 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.939 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.139947+0000" 2024-09-22T03:22:32.939 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.939 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.939 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:22:32.939 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.143247+0000" 2024-09-22T03:22:32.939 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.939 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.939 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.939 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.939 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.939 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.940 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:22:32.940 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.940 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.940 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.940 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.940 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.940 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.940 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.940 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:22:32.940 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.174545+0000" 2024-09-22T03:22:32.940 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.940 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.941 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.941 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.941 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.941 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.941 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:22:32.941 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.941 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:22:32.941 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:22:32.941 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.941 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-22T03:22:32.941 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "client_request", 2024-09-22T03:22:32.941 INFO:teuthology.orchestra.run.smithi153.stdout: "id": 708, 2024-09-22T03:22:32.942 INFO:teuthology.orchestra.run.smithi153.stdout: "detail": { 2024-09-22T03:22:32.942 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.942 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:22:32.942 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.143905+0000" 2024-09-22T03:22:32.942 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.942 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.942 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:22:32.942 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.143916+0000" 2024-09-22T03:22:32.942 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.942 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.942 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.943 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.943 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.943 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.943 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:22:32.943 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.144038+0000" 2024-09-22T03:22:32.943 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.943 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.943 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.943 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.943 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.943 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.943 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:22:32.943 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.944 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.944 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.944 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.944 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.944 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.944 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.944 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:22:32.944 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.144136+0000" 2024-09-22T03:22:32.944 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.944 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.944 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.944 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.944 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.945 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.945 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:22:32.945 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.144932+0000" 2024-09-22T03:22:32.945 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.945 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.945 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.945 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.945 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.945 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.945 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:22:32.945 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.945 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.946 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.946 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.946 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.946 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.946 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.946 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:22:32.946 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.177600+0000" 2024-09-22T03:22:32.946 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.946 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.947 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:22:32.947 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.145304+0000" 2024-09-22T03:22:32.947 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.947 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.947 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.947 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.947 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.947 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.947 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:22:32.947 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.947 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.947 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.948 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.948 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.948 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.948 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.948 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:22:32.948 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.145491+0000" 2024-09-22T03:22:32.948 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.948 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.948 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.948 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.948 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.949 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.949 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:22:32.949 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.949 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.949 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.949 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.949 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.949 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.949 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.949 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:22:32.949 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.145667+0000" 2024-09-22T03:22:32.950 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.950 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.950 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.950 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.950 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.950 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.950 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:22:32.950 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.950 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.950 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.950 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.950 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.951 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.951 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.951 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:22:32.951 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.951 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.951 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.951 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.951 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.951 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.951 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.951 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:22:32.951 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.952 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.952 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.952 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.952 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.952 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.952 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.952 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:22:32.952 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.145941+0000" 2024-09-22T03:22:32.952 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.952 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.952 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.952 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.953 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.953 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.953 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:22:32.953 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.953 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.953 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.953 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.953 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.146386+0000" 2024-09-22T03:22:32.953 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.953 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.953 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:22:32.953 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.147000+0000" 2024-09-22T03:22:32.953 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.954 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.954 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.954 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.954 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.954 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.954 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:22:32.954 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.954 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.954 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.954 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.954 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.954 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.955 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.955 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:22:32.955 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.177357+0000" 2024-09-22T03:22:32.955 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.955 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.955 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.955 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.955 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.955 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.955 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:22:32.955 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.956 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:22:32.956 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:22:32.956 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.956 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-22T03:22:32.956 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "client_request", 2024-09-22T03:22:32.956 INFO:teuthology.orchestra.run.smithi153.stdout: "id": 711, 2024-09-22T03:22:32.956 INFO:teuthology.orchestra.run.smithi153.stdout: "detail": { 2024-09-22T03:22:32.956 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.956 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:22:32.956 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.201296+0000" 2024-09-22T03:22:32.956 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.957 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.957 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:22:32.957 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.201312+0000" 2024-09-22T03:22:32.957 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.957 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.957 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.957 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.957 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.957 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.957 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:22:32.957 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.201469+0000" 2024-09-22T03:22:32.957 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.958 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.958 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.958 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.958 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.958 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.958 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:22:32.958 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.958 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.958 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.959 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.959 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.959 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.959 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.959 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:22:32.959 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.201588+0000" 2024-09-22T03:22:32.959 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.959 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.959 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.959 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.959 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.960 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.960 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:22:32.960 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.209007+0000" 2024-09-22T03:22:32.960 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.960 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.960 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.960 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.960 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.960 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.960 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:22:32.960 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.961 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.961 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.961 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.961 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.961 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.961 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.961 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:22:32.961 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.241353+0000" 2024-09-22T03:22:32.961 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.961 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.961 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:22:32.961 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.209553+0000" 2024-09-22T03:22:32.962 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.962 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.962 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.962 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.962 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.962 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.962 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:22:32.962 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.962 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.962 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.962 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.963 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.963 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.963 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.963 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:22:32.963 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.210822+0000" 2024-09-22T03:22:32.963 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.963 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.963 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.963 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.963 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.963 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.963 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:22:32.964 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.964 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.964 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.964 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.964 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.964 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.964 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.964 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:22:32.964 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.211378+0000" 2024-09-22T03:22:32.964 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.964 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.965 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.965 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.965 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.965 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.965 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:22:32.965 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.965 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.965 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.965 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.965 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.965 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.966 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.966 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:22:32.966 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.966 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.966 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.966 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.966 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.966 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.966 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.966 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:22:32.966 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.966 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.967 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.967 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.967 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.967 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.967 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.967 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:22:32.967 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.211800+0000" 2024-09-22T03:22:32.967 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.967 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.967 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.967 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.967 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.968 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.968 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:22:32.968 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.968 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.968 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.968 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.968 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.968 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.968 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.968 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:22:32.968 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.215672+0000" 2024-09-22T03:22:32.968 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.968 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.969 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.969 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.969 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.969 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.969 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:22:32.969 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.969 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.969 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.969 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.969 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.969 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.969 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.970 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:22:32.970 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.239611+0000" 2024-09-22T03:22:32.970 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.970 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.970 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.970 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.970 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.970 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.970 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:22:32.970 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.970 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:22:32.970 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:22:32.971 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.971 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-22T03:22:32.971 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "client_request", 2024-09-22T03:22:32.971 INFO:teuthology.orchestra.run.smithi153.stdout: "id": 553, 2024-09-22T03:22:32.971 INFO:teuthology.orchestra.run.smithi153.stdout: "detail": { 2024-09-22T03:22:32.971 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.971 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:22:32.971 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.356205+0000" 2024-09-22T03:22:32.971 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.971 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.971 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:22:32.971 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.356211+0000" 2024-09-22T03:22:32.972 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.972 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.972 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.972 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.972 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.972 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.972 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:22:32.972 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.356597+0000" 2024-09-22T03:22:32.972 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.972 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.972 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.972 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.973 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.973 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.973 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:22:32.973 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.973 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.973 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.973 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.973 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.973 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.973 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.973 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:22:32.973 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.356739+0000" 2024-09-22T03:22:32.974 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.974 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.974 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.974 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.974 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.974 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.974 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:22:32.974 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.358825+0000" 2024-09-22T03:22:32.974 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.974 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.974 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.974 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.975 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.975 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.975 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:22:32.975 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.975 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.975 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.975 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.975 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.975 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.975 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.975 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:22:32.975 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.397066+0000" 2024-09-22T03:22:32.976 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.976 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.976 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:22:32.976 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.359236+0000" 2024-09-22T03:22:32.976 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.976 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.976 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.976 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.976 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.976 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.976 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:22:32.976 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.977 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.977 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.977 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.977 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.977 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.977 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.977 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:22:32.977 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.359472+0000" 2024-09-22T03:22:32.977 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.977 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.977 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.977 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.978 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.978 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.978 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:22:32.978 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.978 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.978 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.978 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.978 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.978 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.978 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.978 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:22:32.978 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.359629+0000" 2024-09-22T03:22:32.979 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.979 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.979 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.979 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.979 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.979 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.979 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:22:32.979 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.979 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.979 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.979 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.979 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.980 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.980 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.980 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:22:32.980 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.980 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.980 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.980 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.980 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.980 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.980 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.980 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:22:32.981 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.981 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.981 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.981 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.981 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.981 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.981 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.981 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:22:32.981 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.359797+0000" 2024-09-22T03:22:32.981 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.981 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.981 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.982 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.982 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.982 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.982 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:22:32.982 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.982 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.982 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.982 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.982 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.362697+0000" 2024-09-22T03:22:32.982 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.982 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.982 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:22:32.983 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.373956+0000" 2024-09-22T03:22:32.983 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.983 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.983 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.983 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.983 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.983 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.983 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:22:32.983 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.983 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.983 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.983 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.984 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.984 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.984 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.984 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:22:32.984 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.396833+0000" 2024-09-22T03:22:32.984 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.984 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.984 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.984 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.984 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.984 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.985 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:22:32.985 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.985 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:22:32.985 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:22:32.985 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.985 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-22T03:22:32.985 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "client_request", 2024-09-22T03:22:32.985 INFO:teuthology.orchestra.run.smithi153.stdout: "id": 554, 2024-09-22T03:22:32.985 INFO:teuthology.orchestra.run.smithi153.stdout: "detail": { 2024-09-22T03:22:32.985 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.985 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:22:32.985 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.362669+0000" 2024-09-22T03:22:32.986 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.986 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.986 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:22:32.986 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.362675+0000" 2024-09-22T03:22:32.986 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.986 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.986 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.986 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.986 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.986 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.986 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:22:32.986 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.362749+0000" 2024-09-22T03:22:32.987 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.987 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.987 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.987 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.987 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.987 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.987 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:22:32.987 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.987 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.987 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.987 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.987 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.988 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.988 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.988 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:22:32.988 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.362824+0000" 2024-09-22T03:22:32.988 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.988 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.988 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.988 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.988 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.988 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.988 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:22:32.988 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.365274+0000" 2024-09-22T03:22:32.989 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.989 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.989 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.989 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.989 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.989 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.989 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:22:32.989 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.989 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.989 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.989 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.989 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.989 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.990 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.990 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:22:32.990 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.402847+0000" 2024-09-22T03:22:32.990 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.990 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.990 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:22:32.990 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.365671+0000" 2024-09-22T03:22:32.990 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.990 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.990 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.990 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.990 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.991 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.991 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:22:32.991 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.991 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.991 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.991 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.991 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.991 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.991 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.991 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:22:32.991 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.366038+0000" 2024-09-22T03:22:32.991 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.992 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.992 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.992 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.992 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.992 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.992 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:22:32.992 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.992 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.992 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.992 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.992 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.992 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.993 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.993 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:22:32.993 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.366230+0000" 2024-09-22T03:22:32.993 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.993 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.993 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.993 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.993 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.993 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.993 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:22:32.993 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.993 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.994 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.994 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.994 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.994 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.994 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.994 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:22:32.994 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.994 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.994 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.994 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.994 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.994 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.995 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.995 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:22:32.995 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.995 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.995 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.995 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.995 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.995 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.995 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.995 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:22:32.995 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.366712+0000" 2024-09-22T03:22:32.995 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.996 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.996 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.996 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.996 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.996 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.996 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:22:32.996 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.996 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.996 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.996 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.996 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.367568+0000" 2024-09-22T03:22:32.996 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.997 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.997 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:22:32.997 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.376630+0000" 2024-09-22T03:22:32.997 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.997 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.997 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.997 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.997 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.997 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.997 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:22:32.997 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.997 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.997 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.998 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.998 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.998 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.998 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:32.998 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:22:32.998 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.402579+0000" 2024-09-22T03:22:32.998 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.998 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.998 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:32.998 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.998 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.999 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:32.999 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:22:32.999 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:32.999 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:22:32.999 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:22:32.999 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:32.999 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-22T03:22:32.999 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "client_request", 2024-09-22T03:22:32.999 INFO:teuthology.orchestra.run.smithi153.stdout: "id": 555, 2024-09-22T03:22:32.999 INFO:teuthology.orchestra.run.smithi153.stdout: "detail": { 2024-09-22T03:22:32.999 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.000 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:22:33.000 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.363680+0000" 2024-09-22T03:22:33.000 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.000 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.000 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:22:33.000 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.363685+0000" 2024-09-22T03:22:33.000 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.000 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.000 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.000 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.000 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.000 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.001 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:22:33.001 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.363990+0000" 2024-09-22T03:22:33.001 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.001 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.001 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.001 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.001 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.001 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.001 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:22:33.001 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.001 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.001 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.002 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.002 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.002 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.002 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.002 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:22:33.002 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.364048+0000" 2024-09-22T03:22:33.002 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.002 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.002 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.002 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.002 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.003 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.003 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:22:33.003 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.365356+0000" 2024-09-22T03:22:33.003 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.003 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.003 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.003 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.003 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.003 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.003 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:22:33.003 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.004 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.004 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.004 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.004 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.004 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.004 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.004 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:22:33.004 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.418777+0000" 2024-09-22T03:22:33.004 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.004 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.004 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:22:33.004 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.365897+0000" 2024-09-22T03:22:33.005 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.005 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.005 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.005 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.005 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.005 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.005 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:22:33.005 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.005 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.005 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.005 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.005 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.006 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.006 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.006 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:22:33.006 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.366365+0000" 2024-09-22T03:22:33.006 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.006 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.006 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.006 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.006 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.006 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.006 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:22:33.007 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.007 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.007 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.007 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.007 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.007 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.007 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.007 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:22:33.007 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.367491+0000" 2024-09-22T03:22:33.007 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.007 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.007 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.007 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.008 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.008 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.008 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:22:33.008 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.008 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.008 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.008 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.008 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.008 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.008 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.008 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:22:33.009 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.009 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.009 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.009 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.009 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.009 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.009 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.009 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:22:33.009 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.009 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.009 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.009 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.009 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.010 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.010 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.010 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:22:33.010 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.368014+0000" 2024-09-22T03:22:33.010 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.010 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.010 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.010 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.010 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.010 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.010 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:22:33.011 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.011 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.011 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.011 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.011 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.368326+0000" 2024-09-22T03:22:33.011 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.011 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.011 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:22:33.011 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.380905+0000" 2024-09-22T03:22:33.011 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.011 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.012 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.012 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.012 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.012 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.012 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:22:33.012 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.012 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.013 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.013 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.013 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.013 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.013 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.013 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:22:33.013 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.418497+0000" 2024-09-22T03:22:33.013 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.013 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.013 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.013 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.014 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.014 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.014 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:22:33.014 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.014 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:22:33.014 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:22:33.014 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.015 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-22T03:22:33.015 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "client_request", 2024-09-22T03:22:33.015 INFO:teuthology.orchestra.run.smithi153.stdout: "id": 666, 2024-09-22T03:22:33.015 INFO:teuthology.orchestra.run.smithi153.stdout: "detail": { 2024-09-22T03:22:33.015 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.015 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:22:33.015 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.886858+0000" 2024-09-22T03:22:33.016 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.016 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.016 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:22:33.016 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.886864+0000" 2024-09-22T03:22:33.016 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.016 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.016 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.016 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.016 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.017 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.017 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:22:33.017 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.886953+0000" 2024-09-22T03:22:33.017 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.017 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.018 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.018 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.018 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.018 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.018 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:22:33.018 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.018 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.018 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.018 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.018 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.019 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.019 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.019 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:22:33.019 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.887002+0000" 2024-09-22T03:22:33.019 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.019 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.019 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.019 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.019 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.019 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.019 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:22:33.020 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.887277+0000" 2024-09-22T03:22:33.020 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.020 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.020 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.020 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.020 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.020 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.020 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:22:33.020 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.020 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.020 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.020 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.020 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.021 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.021 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.021 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:22:33.021 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.927674+0000" 2024-09-22T03:22:33.021 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.021 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.021 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:22:33.021 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.887484+0000" 2024-09-22T03:22:33.021 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.021 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.021 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.021 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.022 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.022 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.022 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:22:33.022 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.022 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.022 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.022 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.022 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.022 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.022 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.022 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:22:33.023 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.887581+0000" 2024-09-22T03:22:33.023 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.023 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.023 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.023 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.023 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.023 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.023 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:22:33.023 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.023 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.023 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.024 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.024 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.024 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.024 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.024 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:22:33.024 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.887672+0000" 2024-09-22T03:22:33.024 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.024 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.024 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.024 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.024 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.024 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.024 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:22:33.025 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.025 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.025 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.025 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.025 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.025 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.025 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.025 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:22:33.025 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.025 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.025 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.025 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.026 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.026 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.026 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.026 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:22:33.026 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.026 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.026 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.026 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.026 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.026 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.026 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.026 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:22:33.026 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.887822+0000" 2024-09-22T03:22:33.027 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.027 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.027 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.027 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.027 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.027 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.027 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:22:33.027 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.027 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.027 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.027 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.027 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.028 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.028 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.028 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:22:33.028 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.888535+0000" 2024-09-22T03:22:33.028 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.028 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.028 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.028 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.028 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.028 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.028 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:22:33.028 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.029 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.029 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.029 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.029 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.029 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.029 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.029 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:22:33.029 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.925805+0000" 2024-09-22T03:22:33.029 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.029 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.029 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.029 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.030 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.030 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.030 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:22:33.030 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.030 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:22:33.030 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:22:33.030 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.030 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-22T03:22:33.030 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "client_request", 2024-09-22T03:22:33.030 INFO:teuthology.orchestra.run.smithi153.stdout: "id": 669, 2024-09-22T03:22:33.030 INFO:teuthology.orchestra.run.smithi153.stdout: "detail": { 2024-09-22T03:22:33.030 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.031 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:22:33.031 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.893879+0000" 2024-09-22T03:22:33.031 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.031 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.031 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:22:33.031 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.893886+0000" 2024-09-22T03:22:33.031 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.031 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.031 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.031 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.031 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.031 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.032 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:22:33.032 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.893990+0000" 2024-09-22T03:22:33.032 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.032 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.032 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.032 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.032 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.032 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.032 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:22:33.032 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.032 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.033 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.033 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.033 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.033 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.033 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.033 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:22:33.033 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.894102+0000" 2024-09-22T03:22:33.033 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.033 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.033 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.033 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.033 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.033 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.034 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:22:33.034 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.894378+0000" 2024-09-22T03:22:33.034 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.034 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.034 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.034 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.034 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.034 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.034 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:22:33.034 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.034 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.034 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.035 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.035 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.035 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.035 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.035 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:22:33.035 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.928291+0000" 2024-09-22T03:22:33.035 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.035 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.035 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:22:33.035 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.894655+0000" 2024-09-22T03:22:33.035 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.035 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.036 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.036 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.036 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.036 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.036 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:22:33.036 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.036 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.036 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.036 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.036 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.036 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.036 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.036 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:22:33.037 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.894776+0000" 2024-09-22T03:22:33.037 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.037 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.037 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.037 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.037 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.037 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.037 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:22:33.037 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.037 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.037 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.037 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.038 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.038 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.038 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.038 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:22:33.038 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.894882+0000" 2024-09-22T03:22:33.038 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.038 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.038 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.038 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.038 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.039 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.039 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:22:33.039 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.039 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.039 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.039 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.039 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.039 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.039 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.039 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:22:33.039 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.040 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.040 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.040 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.040 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.040 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.040 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.040 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:22:33.040 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.040 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.040 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.040 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.040 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.041 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.041 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.041 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:22:33.041 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.895041+0000" 2024-09-22T03:22:33.041 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.041 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.041 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.041 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.041 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.041 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.042 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:22:33.042 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.042 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.042 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.042 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.042 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.042 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.042 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.042 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:22:33.042 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.895350+0000" 2024-09-22T03:22:33.042 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.043 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.043 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.043 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.043 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.043 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.043 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:22:33.043 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.043 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.043 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.044 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.044 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.044 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.044 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.044 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:22:33.044 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.928036+0000" 2024-09-22T03:22:33.044 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.044 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.044 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.044 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.044 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.045 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.045 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:22:33.045 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.045 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:22:33.045 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:22:33.045 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.045 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-22T03:22:33.045 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "client_request", 2024-09-22T03:22:33.045 INFO:teuthology.orchestra.run.smithi153.stdout: "id": 671, 2024-09-22T03:22:33.045 INFO:teuthology.orchestra.run.smithi153.stdout: "detail": { 2024-09-22T03:22:33.045 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.045 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:22:33.046 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.901337+0000" 2024-09-22T03:22:33.046 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.046 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.046 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:22:33.046 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.901346+0000" 2024-09-22T03:22:33.046 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.046 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.046 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.046 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.046 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.046 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.047 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:22:33.047 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.901451+0000" 2024-09-22T03:22:33.047 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.047 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.047 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.047 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.047 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.047 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.047 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:22:33.047 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.047 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.048 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.048 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.048 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.048 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.048 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.048 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:22:33.048 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.901500+0000" 2024-09-22T03:22:33.048 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.048 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.048 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.048 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.049 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.049 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.049 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:22:33.049 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.901634+0000" 2024-09-22T03:22:33.049 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.049 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.049 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.049 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.049 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.049 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.049 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:22:33.049 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.050 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.050 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.050 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.050 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.050 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.050 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.050 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:22:33.050 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.930625+0000" 2024-09-22T03:22:33.050 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.050 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.050 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:22:33.050 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.901867+0000" 2024-09-22T03:22:33.051 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.051 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.051 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.051 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.051 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.051 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.051 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:22:33.051 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.051 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.051 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.051 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.051 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.052 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.052 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.052 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:22:33.052 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.901987+0000" 2024-09-22T03:22:33.052 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.052 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.052 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.052 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.052 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.052 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.052 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:22:33.052 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.053 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.053 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.053 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.053 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.053 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.053 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.053 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:22:33.053 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.902100+0000" 2024-09-22T03:22:33.053 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.053 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.053 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.053 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.054 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.054 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.054 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:22:33.054 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.054 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.054 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.054 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.054 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.054 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.054 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.054 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:22:33.054 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.055 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.055 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.055 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.055 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.055 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.055 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.055 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:22:33.055 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.055 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.055 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.055 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.056 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.056 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.056 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.056 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:22:33.056 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.902297+0000" 2024-09-22T03:22:33.056 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.056 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.056 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.056 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.056 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.056 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.056 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:22:33.056 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.057 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.057 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.057 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.057 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.057 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.057 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.057 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:22:33.057 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.902576+0000" 2024-09-22T03:22:33.057 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.057 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.057 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.057 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.058 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.058 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.058 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:22:33.058 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.058 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.058 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.058 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.058 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.058 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.058 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.059 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:22:33.059 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.930326+0000" 2024-09-22T03:22:33.059 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.059 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.059 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.059 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.059 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.059 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.059 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:22:33.059 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.059 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:22:33.060 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:22:33.060 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.060 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-22T03:22:33.060 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "client_request", 2024-09-22T03:22:33.060 INFO:teuthology.orchestra.run.smithi153.stdout: "id": 672, 2024-09-22T03:22:33.060 INFO:teuthology.orchestra.run.smithi153.stdout: "detail": { 2024-09-22T03:22:33.060 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.060 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:22:33.060 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.904576+0000" 2024-09-22T03:22:33.060 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.060 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.060 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:22:33.061 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.904582+0000" 2024-09-22T03:22:33.061 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.061 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.061 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.061 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.061 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.061 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.061 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:22:33.061 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.904711+0000" 2024-09-22T03:22:33.061 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.061 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.062 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.062 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.062 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.062 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.062 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:22:33.062 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.062 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.062 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.062 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.062 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.062 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.063 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.063 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:22:33.063 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.904810+0000" 2024-09-22T03:22:33.063 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.063 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.063 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.063 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.063 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.063 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.063 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:22:33.063 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.905039+0000" 2024-09-22T03:22:33.064 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.064 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.064 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.064 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.064 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.064 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.064 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:22:33.064 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.064 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.064 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.064 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.065 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.065 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.065 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.065 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:22:33.065 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.934975+0000" 2024-09-22T03:22:33.065 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.065 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.065 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:22:33.065 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.905418+0000" 2024-09-22T03:22:33.065 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.065 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.065 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.066 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.066 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.066 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.066 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:22:33.066 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.066 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.066 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.066 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.066 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.066 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.067 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.067 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:22:33.067 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.905573+0000" 2024-09-22T03:22:33.067 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.067 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.067 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.067 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.067 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.067 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.067 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:22:33.067 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.068 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.068 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.068 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.068 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.068 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.068 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.068 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:22:33.068 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.905670+0000" 2024-09-22T03:22:33.068 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.068 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.068 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.068 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.069 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.069 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.069 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:22:33.069 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.069 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.069 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.069 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.069 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.069 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.069 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.069 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:22:33.069 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.070 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.070 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.070 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.070 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.070 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.070 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.070 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:22:33.070 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.070 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.070 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.071 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.071 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.071 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.071 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.071 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:22:33.071 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.905814+0000" 2024-09-22T03:22:33.071 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.071 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.071 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.071 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.071 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.071 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.072 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:22:33.072 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.072 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.072 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.072 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.072 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.072 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.072 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.072 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:22:33.072 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.906096+0000" 2024-09-22T03:22:33.072 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.072 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.073 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.073 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.073 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.073 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.073 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:22:33.073 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.073 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.073 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.073 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.073 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.073 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.073 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.074 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:22:33.074 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.934659+0000" 2024-09-22T03:22:33.074 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.074 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.074 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.074 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.074 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.074 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.074 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:22:33.074 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.074 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:22:33.074 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:22:33.075 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.075 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-22T03:22:33.075 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "client_request", 2024-09-22T03:22:33.075 INFO:teuthology.orchestra.run.smithi153.stdout: "id": 673, 2024-09-22T03:22:33.075 INFO:teuthology.orchestra.run.smithi153.stdout: "detail": { 2024-09-22T03:22:33.075 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.075 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:22:33.075 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.905216+0000" 2024-09-22T03:22:33.075 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.075 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.075 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:22:33.076 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.905221+0000" 2024-09-22T03:22:33.076 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.076 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.076 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.076 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.076 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.076 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.076 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:22:33.076 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.905431+0000" 2024-09-22T03:22:33.076 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.076 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.077 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.077 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.077 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.077 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.077 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:22:33.077 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.077 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.077 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.077 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.077 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.077 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.077 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.078 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:22:33.078 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.905830+0000" 2024-09-22T03:22:33.078 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.078 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.078 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.078 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.078 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.078 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.078 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:22:33.078 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.906751+0000" 2024-09-22T03:22:33.078 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.079 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.079 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.079 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.079 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.079 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.079 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:22:33.079 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.079 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.079 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.079 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.079 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.079 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.080 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.080 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:22:33.080 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.937782+0000" 2024-09-22T03:22:33.080 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.080 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.080 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:22:33.080 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.908199+0000" 2024-09-22T03:22:33.080 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.080 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.080 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.080 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.080 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.081 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.081 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:22:33.081 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.081 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.081 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.081 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.081 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.081 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.081 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.081 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:22:33.081 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.908419+0000" 2024-09-22T03:22:33.082 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.082 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.082 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.082 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.082 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.082 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.082 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:22:33.082 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.082 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.082 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.082 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.083 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.083 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.083 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.083 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:22:33.083 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.908600+0000" 2024-09-22T03:22:33.083 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.083 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.083 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.083 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.083 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.083 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.084 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:22:33.084 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.084 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.084 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.084 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.084 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.084 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.084 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.084 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:22:33.084 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.084 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.085 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.085 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.085 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.085 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.085 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.085 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:22:33.085 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.085 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.085 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.085 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.086 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.086 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.086 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.086 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:22:33.086 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.908888+0000" 2024-09-22T03:22:33.086 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.086 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.086 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.086 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.086 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.086 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.087 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:22:33.087 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.087 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.087 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.087 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.087 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.087 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.087 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.087 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:22:33.087 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.909718+0000" 2024-09-22T03:22:33.088 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.088 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.088 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.088 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.088 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.088 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.088 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:22:33.088 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.088 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.088 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.088 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.088 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.089 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.089 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.089 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:22:33.089 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.937562+0000" 2024-09-22T03:22:33.089 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.089 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.089 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.089 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.089 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.089 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.089 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:22:33.090 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.090 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:22:33.090 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:22:33.090 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.090 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-22T03:22:33.090 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "client_request", 2024-09-22T03:22:33.090 INFO:teuthology.orchestra.run.smithi153.stdout: "id": 674, 2024-09-22T03:22:33.090 INFO:teuthology.orchestra.run.smithi153.stdout: "detail": { 2024-09-22T03:22:33.090 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.090 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:22:33.090 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.906823+0000" 2024-09-22T03:22:33.090 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.091 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.091 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:22:33.091 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.906828+0000" 2024-09-22T03:22:33.091 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.091 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.091 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.091 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.091 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.091 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.091 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:22:33.091 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.906951+0000" 2024-09-22T03:22:33.091 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.092 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.092 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.092 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.092 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.092 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.092 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:22:33.092 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.092 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.092 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.092 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.092 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.093 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.093 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.093 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:22:33.093 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.907033+0000" 2024-09-22T03:22:33.093 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.093 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.093 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.093 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.093 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.093 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.093 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:22:33.094 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.908981+0000" 2024-09-22T03:22:33.094 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.094 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.094 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.094 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.094 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.094 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.094 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:22:33.094 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.094 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.094 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.095 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.095 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.095 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.095 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.095 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:22:33.095 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.942450+0000" 2024-09-22T03:22:33.095 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.095 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.095 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:22:33.095 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.909699+0000" 2024-09-22T03:22:33.095 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.095 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.096 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.096 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.096 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.096 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.096 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:22:33.096 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.096 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.096 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.096 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.096 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.096 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.096 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.097 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:22:33.097 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.909816+0000" 2024-09-22T03:22:33.097 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.097 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.097 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.097 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.097 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.097 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.097 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:22:33.097 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.097 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.098 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.098 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.098 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.098 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.098 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.098 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:22:33.098 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.910204+0000" 2024-09-22T03:22:33.098 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.098 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.098 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.098 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.098 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.099 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.099 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:22:33.099 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.099 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.099 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.099 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.099 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.099 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.099 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.099 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:22:33.099 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.099 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.100 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.100 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.100 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.100 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.100 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.100 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:22:33.100 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.100 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.100 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.100 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.100 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.101 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.101 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.101 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:22:33.101 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.910391+0000" 2024-09-22T03:22:33.101 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.101 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.101 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.101 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.101 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.101 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.101 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:22:33.101 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.102 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.102 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.102 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.102 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.910676+0000" 2024-09-22T03:22:33.102 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.102 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.102 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:22:33.102 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.911293+0000" 2024-09-22T03:22:33.102 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.102 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.102 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.102 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.103 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.103 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.103 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:22:33.103 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.103 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.103 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.103 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.103 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.103 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.103 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.103 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:22:33.103 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.942165+0000" 2024-09-22T03:22:33.104 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.104 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.104 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.104 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.104 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.104 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.104 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:22:33.104 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.104 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:22:33.104 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:22:33.104 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.104 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-22T03:22:33.105 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "client_request", 2024-09-22T03:22:33.105 INFO:teuthology.orchestra.run.smithi153.stdout: "id": 675, 2024-09-22T03:22:33.105 INFO:teuthology.orchestra.run.smithi153.stdout: "detail": { 2024-09-22T03:22:33.105 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.105 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:22:33.105 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.908185+0000" 2024-09-22T03:22:33.105 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.105 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.105 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:22:33.105 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.908190+0000" 2024-09-22T03:22:33.105 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.105 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.106 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.106 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.106 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.106 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.106 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:22:33.106 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.908323+0000" 2024-09-22T03:22:33.106 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.106 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.106 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.106 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.106 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.106 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.107 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:22:33.107 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.107 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.107 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.107 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.107 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.107 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.107 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.107 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:22:33.107 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.908371+0000" 2024-09-22T03:22:33.107 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.108 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.108 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.108 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.108 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.108 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.108 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:22:33.108 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.909017+0000" 2024-09-22T03:22:33.108 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.108 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.108 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.108 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.108 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.109 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.109 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:22:33.109 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.109 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.109 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.109 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.109 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.109 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.109 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.109 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:22:33.109 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.952690+0000" 2024-09-22T03:22:33.109 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.110 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.110 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:22:33.110 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.910066+0000" 2024-09-22T03:22:33.110 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.110 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.110 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.110 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.110 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.110 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.110 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:22:33.110 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.111 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.111 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.111 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.111 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.111 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.111 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.111 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:22:33.111 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.910254+0000" 2024-09-22T03:22:33.111 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.111 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.111 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.111 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.112 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.112 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.112 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:22:33.112 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.112 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.112 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.112 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.112 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.112 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.112 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.112 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:22:33.113 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.910756+0000" 2024-09-22T03:22:33.113 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.113 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.113 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.113 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.113 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.113 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.113 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:22:33.113 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.113 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.113 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.113 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.114 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.114 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.114 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.114 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:22:33.114 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.114 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.114 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.114 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.114 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.114 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.115 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.115 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:22:33.115 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.115 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.115 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.115 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.115 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.115 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.115 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.115 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:22:33.115 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.911042+0000" 2024-09-22T03:22:33.116 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.116 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.116 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.116 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.116 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.116 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.116 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:22:33.116 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.116 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.116 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.116 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.116 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.911650+0000" 2024-09-22T03:22:33.117 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.117 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.117 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:22:33.117 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.915042+0000" 2024-09-22T03:22:33.117 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.117 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.117 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.117 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.117 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.117 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.117 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:22:33.118 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.118 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.118 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.118 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.118 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.118 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.118 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.118 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:22:33.118 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.952454+0000" 2024-09-22T03:22:33.118 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.118 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.118 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.119 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.119 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.119 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.119 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:22:33.119 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.119 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:22:33.119 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:22:33.119 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.119 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-22T03:22:33.119 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "client_request", 2024-09-22T03:22:33.119 INFO:teuthology.orchestra.run.smithi153.stdout: "id": 676, 2024-09-22T03:22:33.119 INFO:teuthology.orchestra.run.smithi153.stdout: "detail": { 2024-09-22T03:22:33.120 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.120 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:22:33.120 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.910997+0000" 2024-09-22T03:22:33.120 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.120 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.120 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:22:33.120 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.911002+0000" 2024-09-22T03:22:33.120 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.120 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.120 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.120 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.121 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.121 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.121 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:22:33.121 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.911079+0000" 2024-09-22T03:22:33.121 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.121 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.121 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.121 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.121 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.121 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.121 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:22:33.121 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.122 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.122 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.122 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.122 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.122 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.122 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.122 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:22:33.122 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.911142+0000" 2024-09-22T03:22:33.122 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.122 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.122 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.122 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.123 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.123 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.123 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:22:33.123 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.913087+0000" 2024-09-22T03:22:33.123 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.123 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.123 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.123 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.123 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.123 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.123 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:22:33.123 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.124 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.124 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.124 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.124 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.124 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.124 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.124 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:22:33.124 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.960758+0000" 2024-09-22T03:22:33.124 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.124 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.124 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:22:33.125 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.913497+0000" 2024-09-22T03:22:33.125 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.125 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.125 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.125 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.125 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.125 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.125 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:22:33.125 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.125 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.125 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.125 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.126 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.126 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.126 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.126 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:22:33.126 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.913749+0000" 2024-09-22T03:22:33.126 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.126 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.126 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.126 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.127 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.127 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.127 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:22:33.127 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.127 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.127 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.127 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.127 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.127 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.127 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.128 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:22:33.128 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.913927+0000" 2024-09-22T03:22:33.128 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.128 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.128 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.128 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.128 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.128 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.128 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:22:33.128 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.129 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.129 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.129 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.129 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.129 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.129 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.129 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:22:33.129 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.129 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.129 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.130 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.130 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.130 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.130 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.130 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:22:33.130 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.130 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.130 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.130 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.130 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.130 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.130 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.131 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:22:33.131 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.914226+0000" 2024-09-22T03:22:33.131 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.131 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.131 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.131 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.131 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.131 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.131 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:22:33.131 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.131 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.131 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.132 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.132 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.914995+0000" 2024-09-22T03:22:33.132 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.132 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.132 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:22:33.132 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.917585+0000" 2024-09-22T03:22:33.132 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.132 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.132 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.132 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.132 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.132 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.133 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:22:33.133 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.133 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.133 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.133 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.133 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.133 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.133 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.133 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:22:33.133 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.960145+0000" 2024-09-22T03:22:33.133 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.133 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.134 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.134 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.134 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.134 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.134 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:22:33.134 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.134 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:22:33.134 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:22:33.134 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.134 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-22T03:22:33.134 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "client_request", 2024-09-22T03:22:33.134 INFO:teuthology.orchestra.run.smithi153.stdout: "id": 685, 2024-09-22T03:22:33.135 INFO:teuthology.orchestra.run.smithi153.stdout: "detail": { 2024-09-22T03:22:33.135 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.135 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:22:33.135 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.953307+0000" 2024-09-22T03:22:33.135 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.135 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.135 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:22:33.135 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.953313+0000" 2024-09-22T03:22:33.135 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.135 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.135 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.135 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.136 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.136 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.136 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:22:33.136 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.953390+0000" 2024-09-22T03:22:33.136 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.136 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.136 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.136 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.136 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.136 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.136 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:22:33.137 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.137 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.137 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.137 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.137 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.137 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.137 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.137 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:22:33.137 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.953456+0000" 2024-09-22T03:22:33.137 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.137 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.138 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.138 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.138 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.138 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.138 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:22:33.138 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.953574+0000" 2024-09-22T03:22:33.138 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.138 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.138 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.138 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.138 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.139 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.139 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:22:33.139 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.139 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.139 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.139 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.139 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.139 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.139 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.139 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:22:33.140 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.977549+0000" 2024-09-22T03:22:33.140 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.140 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.140 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:22:33.140 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.953776+0000" 2024-09-22T03:22:33.140 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.140 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.140 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.140 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.140 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.140 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.140 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:22:33.141 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.141 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.141 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.141 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.141 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.141 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.141 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.141 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:22:33.141 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.953880+0000" 2024-09-22T03:22:33.141 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.141 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.142 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.142 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.142 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.142 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.142 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:22:33.142 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.142 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.142 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.142 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.142 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.142 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.143 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.143 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:22:33.143 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.954029+0000" 2024-09-22T03:22:33.143 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.143 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.143 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.143 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.143 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.143 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.143 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:22:33.143 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.144 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.144 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.144 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.144 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.144 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.144 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.144 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:22:33.144 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.144 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.144 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.144 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.145 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.145 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.145 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.145 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:22:33.145 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.145 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.145 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.145 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.145 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.145 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.145 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.145 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:22:33.146 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.954175+0000" 2024-09-22T03:22:33.146 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.146 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.146 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.146 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.146 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.146 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.146 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:22:33.146 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.146 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.146 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.147 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.147 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.147 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.147 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.147 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:22:33.147 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.955203+0000" 2024-09-22T03:22:33.147 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.147 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.147 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.147 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.147 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.148 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.148 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:22:33.148 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.148 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.148 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.148 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.148 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.148 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.148 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.148 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:22:33.148 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.975567+0000" 2024-09-22T03:22:33.148 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.149 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.149 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.149 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.149 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.149 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.149 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:22:33.149 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.149 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:22:33.149 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:22:33.149 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.149 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-22T03:22:33.150 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "client_request", 2024-09-22T03:22:33.150 INFO:teuthology.orchestra.run.smithi153.stdout: "id": 686, 2024-09-22T03:22:33.150 INFO:teuthology.orchestra.run.smithi153.stdout: "detail": { 2024-09-22T03:22:33.150 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.150 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:22:33.150 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.954279+0000" 2024-09-22T03:22:33.150 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.150 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.150 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:22:33.150 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.954285+0000" 2024-09-22T03:22:33.150 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.151 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.151 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.151 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.151 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.151 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.151 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:22:33.151 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.954402+0000" 2024-09-22T03:22:33.151 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.151 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.151 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.151 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.151 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.152 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.152 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:22:33.152 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.152 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.152 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.152 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.152 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.152 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.152 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.152 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:22:33.152 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.954471+0000" 2024-09-22T03:22:33.153 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.153 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.153 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.153 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.153 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.153 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.153 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:22:33.153 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.954632+0000" 2024-09-22T03:22:33.153 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.153 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.154 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.154 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.154 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.154 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.154 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:22:33.154 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.154 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.154 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.154 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.154 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.154 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.155 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.155 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:22:33.155 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.981693+0000" 2024-09-22T03:22:33.155 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.155 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.155 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:22:33.155 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.954834+0000" 2024-09-22T03:22:33.155 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.155 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.155 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.155 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.156 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.156 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.156 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:22:33.156 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.156 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.156 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.156 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.156 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.156 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.156 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.157 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:22:33.157 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.954936+0000" 2024-09-22T03:22:33.157 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.157 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.157 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.157 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.157 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.157 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.157 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:22:33.157 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.157 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.158 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.158 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.158 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.158 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.158 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.158 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:22:33.158 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.955497+0000" 2024-09-22T03:22:33.158 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.158 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.158 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.158 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.159 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.159 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.159 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:22:33.159 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.159 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.159 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.159 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.159 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.159 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.159 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.159 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:22:33.160 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.160 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.160 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.160 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.160 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.160 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.160 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.160 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:22:33.160 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.160 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.160 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.160 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.161 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.161 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.161 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.161 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:22:33.161 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.956867+0000" 2024-09-22T03:22:33.161 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.161 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.161 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.161 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.161 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.161 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.162 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:22:33.162 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.162 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.162 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.162 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.162 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.162 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.162 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.162 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:22:33.162 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.957329+0000" 2024-09-22T03:22:33.162 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.163 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.163 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.163 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.163 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.163 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.163 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:22:33.163 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.163 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.163 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.163 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.163 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.163 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.164 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.164 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:22:33.164 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.980989+0000" 2024-09-22T03:22:33.164 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.164 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.164 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.164 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.164 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.164 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.164 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:22:33.164 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.165 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:22:33.165 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:22:33.165 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.165 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-22T03:22:33.165 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "client_request", 2024-09-22T03:22:33.165 INFO:teuthology.orchestra.run.smithi153.stdout: "id": 687, 2024-09-22T03:22:33.165 INFO:teuthology.orchestra.run.smithi153.stdout: "detail": { 2024-09-22T03:22:33.165 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.165 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:22:33.165 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.960049+0000" 2024-09-22T03:22:33.165 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.165 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.166 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:22:33.166 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.960063+0000" 2024-09-22T03:22:33.166 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.166 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.166 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.166 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.166 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.166 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.166 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:22:33.166 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.960156+0000" 2024-09-22T03:22:33.166 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.167 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.167 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.167 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.167 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.167 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.167 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:22:33.167 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.167 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.167 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.167 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.167 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.167 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.168 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.168 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:22:33.168 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.960210+0000" 2024-09-22T03:22:33.168 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.168 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.168 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.168 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.168 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.168 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.168 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:22:33.168 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.960956+0000" 2024-09-22T03:22:33.168 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.169 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.169 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.169 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.169 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.169 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.169 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:22:33.169 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.169 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.169 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.169 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.169 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.170 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.170 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.170 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:22:33.170 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.983137+0000" 2024-09-22T03:22:33.170 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.170 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.170 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:22:33.170 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.961208+0000" 2024-09-22T03:22:33.170 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.170 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.170 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.170 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.171 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.171 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.171 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:22:33.171 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.171 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.171 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.171 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.171 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.171 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.171 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.171 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:22:33.172 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.961386+0000" 2024-09-22T03:22:33.172 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.172 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.172 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.172 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.172 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.172 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.172 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:22:33.172 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.172 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.172 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.172 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.173 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.173 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.173 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.173 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:22:33.173 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.961529+0000" 2024-09-22T03:22:33.173 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.173 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.173 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.173 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.173 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.173 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.173 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:22:33.174 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.174 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.174 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.174 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.174 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.174 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.174 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.174 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:22:33.174 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.174 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.174 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.174 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.175 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.175 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.175 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.175 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:22:33.175 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.175 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.175 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.175 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.175 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.175 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.175 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.175 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:22:33.176 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.961719+0000" 2024-09-22T03:22:33.176 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.176 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.176 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.176 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.176 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.176 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.176 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:22:33.176 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.176 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.176 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.176 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.177 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.177 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.177 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.177 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:22:33.177 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.962035+0000" 2024-09-22T03:22:33.177 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.177 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.177 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.177 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.177 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.177 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.178 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:22:33.178 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.178 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.178 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.178 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.178 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.178 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.178 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.178 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:22:33.178 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.982581+0000" 2024-09-22T03:22:33.178 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.178 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.179 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.179 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.179 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.179 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.179 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:22:33.179 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.179 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:22:33.179 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:22:33.179 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.179 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-22T03:22:33.179 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "client_request", 2024-09-22T03:22:33.180 INFO:teuthology.orchestra.run.smithi153.stdout: "id": 688, 2024-09-22T03:22:33.180 INFO:teuthology.orchestra.run.smithi153.stdout: "detail": { 2024-09-22T03:22:33.180 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.180 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:22:33.180 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.961520+0000" 2024-09-22T03:22:33.180 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.180 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.180 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:22:33.180 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.961527+0000" 2024-09-22T03:22:33.180 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.180 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.180 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.180 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.181 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.181 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.181 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:22:33.181 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.961604+0000" 2024-09-22T03:22:33.181 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.181 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.181 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.181 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.181 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.181 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.181 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:22:33.182 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.182 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.182 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.182 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.182 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.182 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.182 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.182 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:22:33.182 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.961651+0000" 2024-09-22T03:22:33.182 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.182 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.182 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.183 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.183 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.183 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.183 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:22:33.183 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.962097+0000" 2024-09-22T03:22:33.183 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.183 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.183 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.183 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.183 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.183 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.183 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:22:33.183 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.184 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.184 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.184 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.184 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.184 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.184 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.184 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:22:33.184 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.984472+0000" 2024-09-22T03:22:33.184 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.184 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.184 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:22:33.185 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.962823+0000" 2024-09-22T03:22:33.185 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.185 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.185 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.185 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.185 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.185 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.185 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:22:33.185 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.185 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.185 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.185 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.186 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.186 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.186 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.186 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:22:33.186 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.963859+0000" 2024-09-22T03:22:33.186 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.186 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.186 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.186 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.186 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.186 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.186 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:22:33.187 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.187 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.187 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.187 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.187 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.187 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.187 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.187 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:22:33.187 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.963999+0000" 2024-09-22T03:22:33.187 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.187 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.187 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.188 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.188 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.188 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.188 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:22:33.188 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.188 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.188 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.188 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.188 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.188 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.188 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.188 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:22:33.189 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.189 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.189 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.189 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.189 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.189 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.189 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.189 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:22:33.189 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.189 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.189 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.189 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.189 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.190 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.190 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.190 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:22:33.190 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.964149+0000" 2024-09-22T03:22:33.190 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.190 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.190 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.190 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.190 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.190 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.190 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:22:33.191 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.191 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.191 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.191 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.191 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.191 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.191 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.191 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:22:33.191 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.964412+0000" 2024-09-22T03:22:33.191 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.191 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.191 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.192 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.192 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.192 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.192 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:22:33.192 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.192 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.192 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.192 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.192 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.192 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.192 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.192 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:22:33.193 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.983955+0000" 2024-09-22T03:22:33.193 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.193 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.193 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.193 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.193 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.193 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.193 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:22:33.193 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.193 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:22:33.193 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:22:33.193 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.194 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-22T03:22:33.194 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "client_request", 2024-09-22T03:22:33.194 INFO:teuthology.orchestra.run.smithi153.stdout: "id": 689, 2024-09-22T03:22:33.194 INFO:teuthology.orchestra.run.smithi153.stdout: "detail": { 2024-09-22T03:22:33.194 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.194 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:22:33.194 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.963235+0000" 2024-09-22T03:22:33.194 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.194 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.194 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:22:33.194 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.963241+0000" 2024-09-22T03:22:33.195 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.195 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.195 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.195 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.195 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.195 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.195 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:22:33.195 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.963317+0000" 2024-09-22T03:22:33.195 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.195 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.195 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.195 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.196 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.196 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.196 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:22:33.196 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.196 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.196 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.196 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.196 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.196 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.196 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.196 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:22:33.196 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.963365+0000" 2024-09-22T03:22:33.197 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.197 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.197 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.197 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.197 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.197 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.197 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:22:33.197 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.964475+0000" 2024-09-22T03:22:33.197 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.197 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.197 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.197 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.198 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.198 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.198 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:22:33.198 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.198 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.198 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.198 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.198 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.198 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.198 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.198 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:22:33.199 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.985776+0000" 2024-09-22T03:22:33.199 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.199 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.199 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:22:33.199 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.964717+0000" 2024-09-22T03:22:33.199 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.199 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.199 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.199 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.199 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.199 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.199 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:22:33.200 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.200 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.200 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.200 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.200 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.200 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.200 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.200 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:22:33.200 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.965046+0000" 2024-09-22T03:22:33.200 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.200 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.201 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.201 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.201 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.201 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.201 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:22:33.201 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.201 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.201 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.201 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.201 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.201 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.201 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.202 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:22:33.202 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.965768+0000" 2024-09-22T03:22:33.202 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.202 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.202 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.202 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.202 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.202 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.202 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:22:33.202 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.203 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.203 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.203 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.203 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.203 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.203 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.203 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:22:33.203 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.203 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.203 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.204 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.204 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.204 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.204 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.204 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:22:33.204 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.204 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.204 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.204 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.204 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.205 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.205 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.205 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:22:33.205 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.966794+0000" 2024-09-22T03:22:33.205 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.205 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.205 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.205 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.205 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.205 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.205 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:22:33.206 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.206 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.206 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.206 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.206 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.206 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.206 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.206 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:22:33.206 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.967366+0000" 2024-09-22T03:22:33.206 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.206 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.207 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.207 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.207 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.207 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.207 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:22:33.207 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.207 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.207 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.207 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.207 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.207 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.208 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.208 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:22:33.208 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.985192+0000" 2024-09-22T03:22:33.208 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.208 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.208 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.208 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.208 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.208 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.208 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:22:33.208 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.209 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:22:33.209 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:22:33.209 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.209 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-22T03:22:33.209 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "client_request", 2024-09-22T03:22:33.209 INFO:teuthology.orchestra.run.smithi153.stdout: "id": 690, 2024-09-22T03:22:33.209 INFO:teuthology.orchestra.run.smithi153.stdout: "detail": { 2024-09-22T03:22:33.209 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.209 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:22:33.209 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.965727+0000" 2024-09-22T03:22:33.209 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.210 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.210 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:22:33.210 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.965733+0000" 2024-09-22T03:22:33.210 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.210 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.210 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.210 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.210 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.210 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.210 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:22:33.210 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.965826+0000" 2024-09-22T03:22:33.210 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.211 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.211 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.211 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.211 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.211 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.211 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:22:33.211 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.211 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.211 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.211 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.212 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.212 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.212 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.212 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:22:33.212 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.965904+0000" 2024-09-22T03:22:33.212 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.212 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.212 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.212 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.212 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.212 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.212 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:22:33.213 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.967423+0000" 2024-09-22T03:22:33.213 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.213 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.213 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.213 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.213 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.213 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.213 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:22:33.213 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.213 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.213 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.214 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.214 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.214 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.214 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.214 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:22:33.214 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.988724+0000" 2024-09-22T03:22:33.214 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.214 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.214 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:22:33.214 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.967809+0000" 2024-09-22T03:22:33.214 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.215 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.215 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.215 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.215 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.215 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.215 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:22:33.215 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.215 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.215 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.215 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.215 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.216 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.216 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.216 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:22:33.216 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.968037+0000" 2024-09-22T03:22:33.216 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.216 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.216 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.216 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.216 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.216 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.216 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:22:33.217 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.217 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.217 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.217 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.217 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.217 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.217 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.217 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:22:33.217 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.968191+0000" 2024-09-22T03:22:33.217 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.217 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.217 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.218 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.218 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.218 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.218 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:22:33.218 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.218 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.218 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.218 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.218 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.218 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.218 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.219 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:22:33.219 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.219 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.219 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.219 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.219 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.219 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.219 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.219 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:22:33.219 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.219 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.219 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.220 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.220 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.220 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.220 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.220 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:22:33.220 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.968343+0000" 2024-09-22T03:22:33.220 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.220 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.220 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.220 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.220 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.220 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.220 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:22:33.221 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.221 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.221 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.221 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.221 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.968659+0000" 2024-09-22T03:22:33.221 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.221 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.221 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:22:33.221 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.969071+0000" 2024-09-22T03:22:33.221 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.221 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.222 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.222 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.222 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.222 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.222 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:22:33.222 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.222 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.222 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.222 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.222 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.222 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.222 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.223 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:22:33.223 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.988456+0000" 2024-09-22T03:22:33.223 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.223 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.223 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.223 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.223 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.223 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.223 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:22:33.223 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.224 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:22:33.224 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:22:33.224 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.224 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-22T03:22:33.224 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "client_request", 2024-09-22T03:22:33.224 INFO:teuthology.orchestra.run.smithi153.stdout: "id": 691, 2024-09-22T03:22:33.224 INFO:teuthology.orchestra.run.smithi153.stdout: "detail": { 2024-09-22T03:22:33.224 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.224 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:22:33.224 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.968358+0000" 2024-09-22T03:22:33.224 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.225 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.225 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:22:33.225 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.968367+0000" 2024-09-22T03:22:33.225 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.225 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.225 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.225 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.225 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.225 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.225 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:22:33.225 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.968492+0000" 2024-09-22T03:22:33.226 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.226 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.226 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.226 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.226 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.226 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.226 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:22:33.226 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.226 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.226 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.226 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.226 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.227 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.227 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.227 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:22:33.227 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.968571+0000" 2024-09-22T03:22:33.227 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.227 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.227 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.227 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.227 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.227 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.227 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:22:33.227 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.970325+0000" 2024-09-22T03:22:33.227 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.228 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.228 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.228 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.228 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.228 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.228 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:22:33.228 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.228 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.228 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.228 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.228 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.229 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.229 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.229 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:22:33.229 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.008360+0000" 2024-09-22T03:22:33.229 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.229 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.229 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:22:33.229 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.970776+0000" 2024-09-22T03:22:33.229 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.229 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.229 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.229 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.230 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.230 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.230 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:22:33.230 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.230 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.230 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.230 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.230 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.230 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.230 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.230 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:22:33.230 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.971024+0000" 2024-09-22T03:22:33.231 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.231 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.231 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.231 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.231 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.231 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.231 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:22:33.231 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.231 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.231 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.231 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.231 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.232 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.232 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.232 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:22:33.232 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.971222+0000" 2024-09-22T03:22:33.232 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.232 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.232 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.232 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.232 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.232 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.232 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:22:33.232 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.233 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.233 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.233 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.233 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.233 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.233 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.233 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:22:33.233 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.233 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.233 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.233 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.233 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.234 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.234 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.234 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:22:33.234 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.234 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.234 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.234 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.234 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.234 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.234 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.234 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:22:33.234 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.971516+0000" 2024-09-22T03:22:33.235 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.235 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.235 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.235 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.235 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.235 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.235 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:22:33.235 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.235 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.235 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.235 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.235 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.971909+0000" 2024-09-22T03:22:33.236 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.236 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.236 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:22:33.236 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.972780+0000" 2024-09-22T03:22:33.236 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.236 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.236 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.236 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.236 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.236 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.236 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:22:33.237 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.237 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.237 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.237 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.237 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.237 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.237 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.237 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:22:33.237 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.008062+0000" 2024-09-22T03:22:33.237 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.237 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.238 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.238 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.238 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.238 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.238 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:22:33.238 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.238 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:22:33.238 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:22:33.238 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.238 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-22T03:22:33.238 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "client_request", 2024-09-22T03:22:33.239 INFO:teuthology.orchestra.run.smithi153.stdout: "id": 692, 2024-09-22T03:22:33.239 INFO:teuthology.orchestra.run.smithi153.stdout: "detail": { 2024-09-22T03:22:33.239 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.239 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-09-22T03:22:33.239 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.970092+0000" 2024-09-22T03:22:33.239 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.239 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.239 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-09-22T03:22:33.239 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.970098+0000" 2024-09-22T03:22:33.239 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.240 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.240 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.240 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.240 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.240 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.240 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-09-22T03:22:33.240 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.970174+0000" 2024-09-22T03:22:33.240 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.240 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.240 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.240 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.241 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.241 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.241 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-09-22T03:22:33.241 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.241 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.241 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.241 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.241 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.241 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.241 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.241 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-09-22T03:22:33.242 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.970285+0000" 2024-09-22T03:22:33.242 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.242 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.242 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.242 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.242 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.242 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.242 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-09-22T03:22:33.242 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.971322+0000" 2024-09-22T03:22:33.242 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.242 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.242 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.243 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.243 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.243 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.243 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-09-22T03:22:33.243 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.243 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.243 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.243 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.243 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.243 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.244 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.244 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:22:33.244 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.022056+0000" 2024-09-22T03:22:33.244 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.244 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.244 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::AwaitMap", 2024-09-22T03:22:33.244 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.972090+0000" 2024-09-22T03:22:33.244 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.244 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.244 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.244 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.245 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.245 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.245 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-09-22T03:22:33.245 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.245 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.245 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.245 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::OSDMapBlocker>::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.245 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.245 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.245 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.245 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::WaitForActive", 2024-09-22T03:22:33.246 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.972243+0000" 2024-09-22T03:22:33.246 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.246 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.246 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.246 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.246 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.246 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.246 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::PGActivationBlocker", 2024-09-22T03:22:33.246 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.246 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.246 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.247 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.247 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.247 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.247 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.247 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::RecoverMissing", 2024-09-22T03:22:33.247 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.972382+0000" 2024-09-22T03:22:33.247 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.247 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.247 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.247 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.247 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.248 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.248 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingLockOBC", 2024-09-22T03:22:33.248 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.248 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.248 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.248 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.248 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.248 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.248 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.248 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::RecoverMissingSnaps", 2024-09-22T03:22:33.248 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.248 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.249 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.249 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.249 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.249 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.249 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.249 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::scrub::PGScrubber", 2024-09-22T03:22:33.249 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.249 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.249 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.249 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.250 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.250 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.250 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.250 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::GetOBC", 2024-09-22T03:22:33.250 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.972824+0000" 2024-09-22T03:22:33.250 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.250 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.250 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.250 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.250 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.250 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.251 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::LockOBC", 2024-09-22T03:22:33.251 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.251 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.251 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.251 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.251 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.973143+0000" 2024-09-22T03:22:33.251 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.251 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.251 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::CommonPGPipeline::Process", 2024-09-22T03:22:33.251 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:31.975230+0000" 2024-09-22T03:22:33.251 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.251 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.252 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.252 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.252 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.252 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.252 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::WaitRepop", 2024-09-22T03:22:33.252 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.252 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.252 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.252 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.252 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.252 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.253 INFO:teuthology.orchestra.run.smithi153.stdout: "blocking_event": { 2024-09-22T03:22:33.253 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::ClientRequest::PGPipeline::SendReply", 2024-09-22T03:22:33.253 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-09-22T03:22:32.021731+0000" 2024-09-22T03:22:33.253 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.253 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.253 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::BlockerT::BlockingEvent::ExitBarrierEvent", 2024-09-22T03:22:33.253 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.253 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-22T03:22:33.253 INFO:teuthology.orchestra.run.smithi153.stdout: "time_event": { 2024-09-22T03:22:33.253 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-09-22T03:22:33.253 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "0.000000" 2024-09-22T03:22:33.253 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:22:33.254 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:22:33.254 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-22T03:22:33.254 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-22T03:22:33.254 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-22T03:22:33.255 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 79 16 3436 3420 0.338171 2.70312 0.0409324 0.369546 2024-09-22T03:22:33.855 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-22T03:22:33.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:22:33.985+0000 7f2879397640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:22:33.993 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:22:33.992+0000 7f2879397640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:22:34.075 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:2024-09-22T03:22:34.074698+0000 min lat: 0.00801997 max lat: 68.3554 avg lat: 0.342298 2024-09-22T03:22:34.075 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-22T03:22:34.075 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 80 16 3754 3738 0.364995 2.48438 0.0360504 0.342298 2024-09-22T03:22:35.075 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 81 16 4104 4088 0.394242 2.73438 0.031473 0.316962 2024-09-22T03:22:36.075 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 82 16 4446 4430 0.422015 2.67188 0.0600125 0.296104 2024-09-22T03:22:36.173 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-22T03:22:36.321 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:22:36.317+0000 7f0f2b86c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:22:36.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:22:36.324+0000 7f0f2b86c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:22:37.075 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 83 16 4788 4772 0.449118 2.67188 0.035465 0.278251 2024-09-22T03:22:38.075 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 84 16 5137 5121 0.476227 2.72656 0.08507 0.262339 2024-09-22T03:22:38.256 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [1, 3, 2, 0] 2024-09-22T03:22:38.256 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-09-22T03:22:38.396 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:22:38.387+0000 7f019c858640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:22:38.401 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:22:38.399+0000 7f019c858640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:22:38.472 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-22T03:22:38.607 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:22:38.604+0000 7f09ef62c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:22:38.612 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:22:38.611+0000 7f09ef62c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:22:39.005 INFO:teuthology.orchestra.run.smithi153.stderr:reweighted osd.0 to 1 (10000) 2024-09-22T03:22:39.028 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-09-22T03:22:39.075 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 85 16 5415 5399 0.496173 2.17188 0.0279782 0.251862 2024-09-22T03:22:39.168 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:22:39.158+0000 7fec24b17640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:22:39.174 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:22:39.172+0000 7fec24b17640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:22:40.012 INFO:teuthology.orchestra.run.smithi153.stderr:reweighted osd.1 to 1 (10000) 2024-09-22T03:22:40.035 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-09-22T03:22:40.076 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 86 16 5753 5737 0.521105 2.64062 0.0308269 0.239765 2024-09-22T03:22:40.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:22:40.160+0000 7fe9f0397640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:22:40.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:22:40.176+0000 7fe9f0397640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:22:40.840 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-22T03:22:40.981 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:22:40.978+0000 7fbb19117640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:22:40.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:22:40.984+0000 7fbb19117640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:22:41.019 INFO:teuthology.orchestra.run.smithi153.stderr:reweighted osd.2 to 1 (10000) 2024-09-22T03:22:41.042 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-09-22T03:22:41.075 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 87 16 6068 6052 0.543398 2.46094 0.0294635 0.229943 2024-09-22T03:22:41.189 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:22:41.180+0000 7f0cde397640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:22:41.193 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:22:41.192+0000 7f0cde397640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:22:42.023 INFO:teuthology.orchestra.run.smithi153.stderr:reweighted osd.3 to 1 (10000) 2024-09-22T03:22:42.043 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-09-22T03:22:42.043 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-22T03:22:42.075 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 88 16 6396 6380 0.566339 2.5625 0.07796 0.22062 2024-09-22T03:22:42.184 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:22:42.177+0000 7f4e514ba640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:22:42.191 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:22:42.190+0000 7f4e514ba640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:22:42.457 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-22T03:22:42.457 INFO:teuthology.orchestra.run.smithi153.stderr:dumped all 2024-09-22T03:22:42.475 INFO:teuthology.orchestra.run.smithi153.stdout:{"pg_ready":true,"pg_map":{"version":601,"stamp":"2024-09-22T03:22:42.293468+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":44950067,"num_objects":682,"num_object_clones":0,"num_object_copies":2046,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":5554,"num_write_kb":45216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":112,"num_bytes_recovered":3170304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5504,"ondisk_log_size":5504,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":188328,"kb_used_data":67112,"kb_used_omap":0,"kb_used_meta":121152,"kb_avail":377299032,"statfs":{"total":386547056640,"available":386354208768,"internally_reserved":0,"allocated":68722688,"data_stored":136327980,"data_compressed":6890422,"data_compressed_allocated":63119360,"data_compressed_original":134029312,"omap_allocated":0,"internal_metadata":124059648},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":10338304,"num_objects":158,"num_object_clones":0,"num_object_copies":474,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1262,"num_write_kb":10096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000951"},"pg_stats":[{"pgid":"10.7","version":"165'367","reported_seq":327,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-22T03:22:39.330521+0000","last_change":"2024-09-22T03:22:30.549286+0000","last_active":"2024-09-22T03:22:39.330521+0000","last_peered":"2024-09-22T03:22:39.330521+0000","last_clean":"2024-09-22T03:22:39.330521+0000","last_became_active":"2024-09-22T03:22:30.085182+0000","last_became_peered":"2024-09-22T03:22:30.085182+0000","last_unstale":"2024-09-22T03:22:39.330521+0000","last_undegraded":"2024-09-22T03:22:39.330521+0000","last_fullsized":"2024-09-22T03:22:39.330521+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:21:11.969997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:21:11.969997+0000","last_clean_scrub_stamp":"2024-09-22T03:21:11.969997+0000","objects_scrubbed":0,"log_size":367,"log_dups_size":0,"ondisk_log_size":367,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3014656,"num_objects":46,"num_object_clones":0,"num_object_copies":138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":368,"num_write_kb":2944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.6","version":"164'351","reported_seq":415,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-22T03:22:37.691439+0000","last_change":"2024-09-22T03:22:30.891241+0000","last_active":"2024-09-22T03:22:37.691439+0000","last_peered":"2024-09-22T03:22:37.691439+0000","last_clean":"2024-09-22T03:22:37.691439+0000","last_became_active":"2024-09-22T03:22:30.100277+0000","last_became_peered":"2024-09-22T03:22:30.100277+0000","last_unstale":"2024-09-22T03:22:37.691439+0000","last_undegraded":"2024-09-22T03:22:37.691439+0000","last_fullsized":"2024-09-22T03:22:37.691439+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:21:11.969997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:21:11.969997+0000","last_clean_scrub_stamp":"2024-09-22T03:21:11.969997+0000","objects_scrubbed":0,"log_size":351,"log_dups_size":0,"ondisk_log_size":351,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2883584,"num_objects":44,"num_object_clones":0,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":352,"num_write_kb":2816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":524288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.5","version":"165'416","reported_seq":457,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-22T03:22:40.025212+0000","last_change":"2024-09-22T03:22:16.387575+0000","last_active":"2024-09-22T03:22:40.025212+0000","last_peered":"2024-09-22T03:22:40.025212+0000","last_clean":"2024-09-22T03:22:40.025212+0000","last_became_active":"2024-09-22T03:22:16.383638+0000","last_became_peered":"2024-09-22T03:22:16.383638+0000","last_unstale":"2024-09-22T03:22:40.025212+0000","last_undegraded":"2024-09-22T03:22:40.025212+0000","last_fullsized":"2024-09-22T03:22:40.025212+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:21:11.969997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:21:11.969997+0000","last_clean_scrub_stamp":"2024-09-22T03:21:11.969997+0000","objects_scrubbed":0,"log_size":416,"log_dups_size":0,"ondisk_log_size":416,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3407872,"num_objects":52,"num_object_clones":0,"num_object_copies":156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":416,"num_write_kb":3328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.4","version":"165'319","reported_seq":365,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-22T03:22:39.050225+0000","last_change":"2024-09-22T03:22:16.388149+0000","last_active":"2024-09-22T03:22:39.050225+0000","last_peered":"2024-09-22T03:22:39.050225+0000","last_clean":"2024-09-22T03:22:39.050225+0000","last_became_active":"2024-09-22T03:22:16.384355+0000","last_became_peered":"2024-09-22T03:22:16.384355+0000","last_unstale":"2024-09-22T03:22:39.050225+0000","last_undegraded":"2024-09-22T03:22:39.050225+0000","last_fullsized":"2024-09-22T03:22:39.050225+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:21:11.969997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:21:11.969997+0000","last_clean_scrub_stamp":"2024-09-22T03:21:11.969997+0000","objects_scrubbed":0,"log_size":319,"log_dups_size":0,"ondisk_log_size":319,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2621440,"num_objects":40,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":320,"num_write_kb":2560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.3","version":"165'351","reported_seq":339,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-22T03:22:39.240776+0000","last_change":"2024-09-22T03:22:31.459047+0000","last_active":"2024-09-22T03:22:39.240776+0000","last_peered":"2024-09-22T03:22:39.240776+0000","last_clean":"2024-09-22T03:22:39.240776+0000","last_became_active":"2024-09-22T03:22:30.079088+0000","last_became_peered":"2024-09-22T03:22:30.079088+0000","last_unstale":"2024-09-22T03:22:39.240776+0000","last_undegraded":"2024-09-22T03:22:39.240776+0000","last_fullsized":"2024-09-22T03:22:39.240776+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:21:11.969997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:21:11.969997+0000","last_clean_scrub_stamp":"2024-09-22T03:21:11.969997+0000","objects_scrubbed":0,"log_size":351,"log_dups_size":0,"ondisk_log_size":351,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2883584,"num_objects":44,"num_object_clones":0,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":352,"num_write_kb":2816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":8192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.2","version":"164'320","reported_seq":383,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-22T03:22:39.012701+0000","last_change":"2024-09-22T03:22:30.332579+0000","last_active":"2024-09-22T03:22:39.012701+0000","last_peered":"2024-09-22T03:22:39.012701+0000","last_clean":"2024-09-22T03:22:39.012701+0000","last_became_active":"2024-09-22T03:22:30.110457+0000","last_became_peered":"2024-09-22T03:22:30.110457+0000","last_unstale":"2024-09-22T03:22:39.012701+0000","last_undegraded":"2024-09-22T03:22:39.012701+0000","last_fullsized":"2024-09-22T03:22:39.012701+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:21:11.969997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:21:11.969997+0000","last_clean_scrub_stamp":"2024-09-22T03:21:11.969997+0000","objects_scrubbed":0,"log_size":320,"log_dups_size":0,"ondisk_log_size":320,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2621440,"num_objects":40,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":320,"num_write_kb":2560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":196608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1","version":"165'367","reported_seq":336,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-22T03:22:39.175565+0000","last_change":"2024-09-22T03:22:30.754141+0000","last_active":"2024-09-22T03:22:39.175565+0000","last_peered":"2024-09-22T03:22:39.175565+0000","last_clean":"2024-09-22T03:22:39.175565+0000","last_became_active":"2024-09-22T03:22:30.109611+0000","last_became_peered":"2024-09-22T03:22:30.109611+0000","last_unstale":"2024-09-22T03:22:39.175565+0000","last_undegraded":"2024-09-22T03:22:39.175565+0000","last_fullsized":"2024-09-22T03:22:39.175565+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:21:11.969997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:21:11.969997+0000","last_clean_scrub_stamp":"2024-09-22T03:21:11.969997+0000","objects_scrubbed":0,"log_size":367,"log_dups_size":0,"ondisk_log_size":367,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3014656,"num_objects":46,"num_object_clones":0,"num_object_copies":138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":368,"num_write_kb":2944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.0","version":"164'311","reported_seq":379,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-22T03:22:37.159536+0000","last_change":"2024-09-22T03:22:31.226107+0000","last_active":"2024-09-22T03:22:37.159536+0000","last_peered":"2024-09-22T03:22:37.159536+0000","last_clean":"2024-09-22T03:22:37.159536+0000","last_became_active":"2024-09-22T03:22:30.105137+0000","last_became_peered":"2024-09-22T03:22:30.105137+0000","last_unstale":"2024-09-22T03:22:37.159536+0000","last_undegraded":"2024-09-22T03:22:37.159536+0000","last_fullsized":"2024-09-22T03:22:37.159536+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:21:11.969997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:21:11.969997+0000","last_clean_scrub_stamp":"2024-09-22T03:21:11.969997+0000","objects_scrubbed":0,"log_size":311,"log_dups_size":0,"ondisk_log_size":311,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2555904,"num_objects":39,"num_object_clones":0,"num_object_copies":117,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":312,"num_write_kb":2496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":655360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.b","version":"164'347","reported_seq":402,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-22T03:22:37.705464+0000","last_change":"2024-09-22T03:22:16.420237+0000","last_active":"2024-09-22T03:22:37.705464+0000","last_peered":"2024-09-22T03:22:37.705464+0000","last_clean":"2024-09-22T03:22:37.705464+0000","last_became_active":"2024-09-22T03:22:16.396604+0000","last_became_peered":"2024-09-22T03:22:16.396604+0000","last_unstale":"2024-09-22T03:22:37.705464+0000","last_undegraded":"2024-09-22T03:22:37.705464+0000","last_fullsized":"2024-09-22T03:22:37.705464+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:21:11.969997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:21:11.969997+0000","last_clean_scrub_stamp":"2024-09-22T03:21:11.969997+0000","objects_scrubbed":0,"log_size":347,"log_dups_size":0,"ondisk_log_size":347,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2850816,"num_objects":44,"num_object_clones":0,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":348,"num_write_kb":2784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":233,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-22T03:22:39.013931+0000","last_change":"2024-09-22T03:22:39.015859+0000","last_active":"2024-09-22T03:22:39.013931+0000","last_peered":"2024-09-22T03:22:39.013931+0000","last_clean":"2024-09-22T03:22:39.013931+0000","last_became_active":"2024-09-22T03:22:30.096861+0000","last_became_peered":"2024-09-22T03:22:30.096861+0000","last_unstale":"2024-09-22T03:22:39.013931+0000","last_undegraded":"2024-09-22T03:22:39.013931+0000","last_fullsized":"2024-09-22T03:22:39.013931+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.a","version":"165'343","reported_seq":330,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-22T03:22:39.292396+0000","last_change":"2024-09-22T03:22:31.215850+0000","last_active":"2024-09-22T03:22:39.292396+0000","last_peered":"2024-09-22T03:22:39.292396+0000","last_clean":"2024-09-22T03:22:39.292396+0000","last_became_active":"2024-09-22T03:22:30.129087+0000","last_became_peered":"2024-09-22T03:22:30.129087+0000","last_unstale":"2024-09-22T03:22:39.292396+0000","last_undegraded":"2024-09-22T03:22:39.292396+0000","last_fullsized":"2024-09-22T03:22:39.292396+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:21:11.969997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:21:11.969997+0000","last_clean_scrub_stamp":"2024-09-22T03:21:11.969997+0000","objects_scrubbed":0,"log_size":343,"log_dups_size":0,"ondisk_log_size":343,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2818048,"num_objects":43,"num_object_clones":0,"num_object_copies":129,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":344,"num_write_kb":2752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":366,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-22T03:22:39.022294+0000","last_change":"2024-09-22T03:22:39.030882+0000","last_active":"2024-09-22T03:22:39.022294+0000","last_peered":"2024-09-22T03:22:39.022294+0000","last_clean":"2024-09-22T03:22:39.022294+0000","last_became_active":"2024-09-22T03:22:30.117777+0000","last_became_peered":"2024-09-22T03:22:30.117777+0000","last_unstale":"2024-09-22T03:22:39.022294+0000","last_undegraded":"2024-09-22T03:22:39.022294+0000","last_fullsized":"2024-09-22T03:22:39.022294+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.9","version":"164'304","reported_seq":379,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-22T03:22:39.012343+0000","last_change":"2024-09-22T03:22:30.438260+0000","last_active":"2024-09-22T03:22:39.012343+0000","last_peered":"2024-09-22T03:22:39.012343+0000","last_clean":"2024-09-22T03:22:39.012343+0000","last_became_active":"2024-09-22T03:22:30.100388+0000","last_became_peered":"2024-09-22T03:22:30.100388+0000","last_unstale":"2024-09-22T03:22:39.012343+0000","last_undegraded":"2024-09-22T03:22:39.012343+0000","last_fullsized":"2024-09-22T03:22:39.012343+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:21:11.969997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:21:11.969997+0000","last_clean_scrub_stamp":"2024-09-22T03:21:11.969997+0000","objects_scrubbed":0,"log_size":304,"log_dups_size":0,"ondisk_log_size":304,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2490368,"num_objects":38,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":304,"num_write_kb":2432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":589824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":397,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-22T03:22:40.027260+0000","last_change":"2024-09-22T03:22:25.014526+0000","last_active":"2024-09-22T03:22:40.027260+0000","last_peered":"2024-09-22T03:22:40.027260+0000","last_clean":"2024-09-22T03:22:40.027260+0000","last_became_active":"2024-09-22T03:22:16.387817+0000","last_became_peered":"2024-09-22T03:22:16.387817+0000","last_unstale":"2024-09-22T03:22:40.027260+0000","last_undegraded":"2024-09-22T03:22:40.027260+0000","last_fullsized":"2024-09-22T03:22:40.027260+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.8","version":"166'303","reported_seq":353,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-22T03:22:40.332304+0000","last_change":"2024-09-22T03:22:30.185137+0000","last_active":"2024-09-22T03:22:40.332304+0000","last_peered":"2024-09-22T03:22:40.332304+0000","last_clean":"2024-09-22T03:22:40.332304+0000","last_became_active":"2024-09-22T03:22:30.053731+0000","last_became_peered":"2024-09-22T03:22:30.053731+0000","last_unstale":"2024-09-22T03:22:40.332304+0000","last_undegraded":"2024-09-22T03:22:40.332304+0000","last_fullsized":"2024-09-22T03:22:40.332304+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:21:11.969997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:21:11.969997+0000","last_clean_scrub_stamp":"2024-09-22T03:21:11.969997+0000","objects_scrubbed":0,"log_size":303,"log_dups_size":0,"ondisk_log_size":303,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2490368,"num_objects":38,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":304,"num_write_kb":2432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":281,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-22T03:22:39.012682+0000","last_change":"2024-09-22T03:22:39.013700+0000","last_active":"2024-09-22T03:22:39.012682+0000","last_peered":"2024-09-22T03:22:39.012682+0000","last_clean":"2024-09-22T03:22:39.012682+0000","last_became_active":"2024-09-22T03:22:30.068029+0000","last_became_peered":"2024-09-22T03:22:30.068029+0000","last_unstale":"2024-09-22T03:22:39.012682+0000","last_undegraded":"2024-09-22T03:22:39.012682+0000","last_fullsized":"2024-09-22T03:22:39.012682+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"12'97","reported_seq":486,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-22T03:22:39.024706+0000","last_change":"2024-09-22T03:22:30.096425+0000","last_active":"2024-09-22T03:22:39.024706+0000","last_peered":"2024-09-22T03:22:39.024706+0000","last_clean":"2024-09-22T03:22:39.024706+0000","last_became_active":"2024-09-22T03:22:30.088734+0000","last_became_peered":"2024-09-22T03:22:30.088734+0000","last_unstale":"2024-09-22T03:22:39.024706+0000","last_undegraded":"2024-09-22T03:22:39.024706+0000","last_fullsized":"2024-09-22T03:22:39.024706+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.f","version":"164'311","reported_seq":383,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-22T03:22:37.638721+0000","last_change":"2024-09-22T03:22:30.515318+0000","last_active":"2024-09-22T03:22:37.638721+0000","last_peered":"2024-09-22T03:22:37.638721+0000","last_clean":"2024-09-22T03:22:37.638721+0000","last_became_active":"2024-09-22T03:22:30.054165+0000","last_became_peered":"2024-09-22T03:22:30.054165+0000","last_unstale":"2024-09-22T03:22:37.638721+0000","last_undegraded":"2024-09-22T03:22:37.638721+0000","last_fullsized":"2024-09-22T03:22:37.638721+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:21:11.969997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:21:11.969997+0000","last_clean_scrub_stamp":"2024-09-22T03:21:11.969997+0000","objects_scrubbed":0,"log_size":311,"log_dups_size":0,"ondisk_log_size":311,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2555904,"num_objects":39,"num_object_clones":0,"num_object_copies":117,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":312,"num_write_kb":2496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":786432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":322,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-22T03:22:30.106450+0000","last_change":"2024-09-22T03:22:30.106962+0000","last_active":"2024-09-22T03:22:30.106450+0000","last_peered":"2024-09-22T03:22:30.106450+0000","last_clean":"2024-09-22T03:21:50.022603+0000","last_became_active":"2024-09-22T03:22:30.102934+0000","last_became_peered":"2024-09-22T03:22:30.102934+0000","last_unstale":"2024-09-22T03:22:30.106450+0000","last_undegraded":"2024-09-22T03:22:30.106450+0000","last_fullsized":"2024-09-22T03:22:30.106450+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.e","version":"165'346","reported_seq":327,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-22T03:22:39.367480+0000","last_change":"2024-09-22T03:22:31.235882+0000","last_active":"2024-09-22T03:22:39.367480+0000","last_peered":"2024-09-22T03:22:39.367480+0000","last_clean":"2024-09-22T03:22:39.367480+0000","last_became_active":"2024-09-22T03:22:30.135157+0000","last_became_peered":"2024-09-22T03:22:30.135157+0000","last_unstale":"2024-09-22T03:22:39.367480+0000","last_undegraded":"2024-09-22T03:22:39.367480+0000","last_fullsized":"2024-09-22T03:22:39.367480+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:21:11.969997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:21:11.969997+0000","last_clean_scrub_stamp":"2024-09-22T03:21:11.969997+0000","objects_scrubbed":0,"log_size":346,"log_dups_size":0,"ondisk_log_size":346,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2842624,"num_objects":44,"num_object_clones":0,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":347,"num_write_kb":2776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":367,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-22T03:22:40.026301+0000","last_change":"2024-09-22T03:22:39.026321+0000","last_active":"2024-09-22T03:22:40.026301+0000","last_peered":"2024-09-22T03:22:40.026301+0000","last_clean":"2024-09-22T03:22:40.026301+0000","last_became_active":"2024-09-22T03:22:30.055276+0000","last_became_peered":"2024-09-22T03:22:30.055276+0000","last_unstale":"2024-09-22T03:22:40.026301+0000","last_undegraded":"2024-09-22T03:22:40.026301+0000","last_fullsized":"2024-09-22T03:22:40.026301+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.d","version":"164'312","reported_seq":292,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-22T03:22:39.031852+0000","last_change":"2024-09-22T03:22:30.910625+0000","last_active":"2024-09-22T03:22:39.031852+0000","last_peered":"2024-09-22T03:22:39.031852+0000","last_clean":"2024-09-22T03:22:39.031852+0000","last_became_active":"2024-09-22T03:22:30.111068+0000","last_became_peered":"2024-09-22T03:22:30.111068+0000","last_unstale":"2024-09-22T03:22:39.031852+0000","last_undegraded":"2024-09-22T03:22:39.031852+0000","last_fullsized":"2024-09-22T03:22:39.031852+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:21:11.969997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:21:11.969997+0000","last_clean_scrub_stamp":"2024-09-22T03:21:11.969997+0000","objects_scrubbed":0,"log_size":312,"log_dups_size":0,"ondisk_log_size":312,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2555904,"num_objects":39,"num_object_clones":0,"num_object_copies":117,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":312,"num_write_kb":2496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":400,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-22T03:22:30.012451+0000","last_change":"2024-09-22T03:22:25.010645+0000","last_active":"2024-09-22T03:22:30.012451+0000","last_peered":"2024-09-22T03:22:30.012451+0000","last_clean":"2024-09-22T03:22:30.012451+0000","last_became_active":"2024-09-22T03:22:16.384462+0000","last_became_peered":"2024-09-22T03:22:16.384462+0000","last_unstale":"2024-09-22T03:22:30.012451+0000","last_undegraded":"2024-09-22T03:22:30.012451+0000","last_fullsized":"2024-09-22T03:22:30.012451+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.c","version":"164'336","reported_seq":316,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-22T03:22:39.027978+0000","last_change":"2024-09-22T03:22:30.382901+0000","last_active":"2024-09-22T03:22:39.027978+0000","last_peered":"2024-09-22T03:22:39.027978+0000","last_clean":"2024-09-22T03:22:39.027978+0000","last_became_active":"2024-09-22T03:22:30.094983+0000","last_became_peered":"2024-09-22T03:22:30.094983+0000","last_unstale":"2024-09-22T03:22:39.027978+0000","last_undegraded":"2024-09-22T03:22:39.027978+0000","last_fullsized":"2024-09-22T03:22:39.027978+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:21:11.969997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:21:11.969997+0000","last_clean_scrub_stamp":"2024-09-22T03:21:11.969997+0000","objects_scrubbed":0,"log_size":336,"log_dups_size":0,"ondisk_log_size":336,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2752512,"num_objects":42,"num_object_clones":0,"num_object_copies":126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":336,"num_write_kb":2688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":16384,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":407,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-22T03:22:30.107638+0000","last_change":"2024-09-22T03:22:30.108468+0000","last_active":"2024-09-22T03:22:30.107638+0000","last_peered":"2024-09-22T03:22:30.107638+0000","last_clean":"2024-09-22T03:22:16.362770+0000","last_became_active":"2024-09-22T03:22:30.101199+0000","last_became_peered":"2024-09-22T03:22:30.101199+0000","last_unstale":"2024-09-22T03:22:30.107638+0000","last_undegraded":"2024-09-22T03:22:30.107638+0000","last_fullsized":"2024-09-22T03:22:30.107638+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":16,"stat_sum":{"num_bytes":44359680,"num_objects":678,"num_object_clones":0,"num_object_copies":2034,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5415,"num_write_kb":43320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":112,"num_bytes_recovered":3170304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":64192512,"data_stored":134701056,"data_compressed":6924052,"data_compressed_allocated":64192512,"data_compressed_original":134701056,"omap_allocated":0,"internal_metadata":0},"log_size":5404,"ondisk_log_size":5404,"up":48,"acting":48,"num_store_stats":4},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1180736,"data_compressed":59924,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":155,"seq":665719930930,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":44244,"kb_used_data":13508,"kb_used_omap":0,"kb_used_meta":30720,"kb_avail":94327596,"statfs":{"total":96636764160,"available":96591458304,"internally_reserved":0,"allocated":13832192,"data_stored":25499242,"data_compressed":1286710,"data_compressed_allocated":12443648,"data_compressed_original":24936448,"omap_allocated":0,"internal_metadata":31457280},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":159,"seq":682899800089,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":55400,"kb_used_data":22104,"kb_used_omap":0,"kb_used_meta":33280,"kb_avail":94316440,"statfs":{"total":96636764160,"available":96580034560,"internally_reserved":0,"allocated":22634496,"data_stored":43541843,"data_compressed":2217450,"data_compressed_allocated":21217280,"data_compressed_original":42958848,"omap_allocated":0,"internal_metadata":34078720},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":138,"seq":592705487037,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":45360,"kb_used_data":15968,"kb_used_omap":0,"kb_used_meta":29376,"kb_avail":94326480,"statfs":{"total":96636764160,"available":96590315520,"internally_reserved":0,"allocated":16351232,"data_stored":30968769,"data_compressed":1568812,"data_compressed_allocated":14950400,"data_compressed_original":30392320,"omap_allocated":0,"internal_metadata":30081024},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":163,"seq":700079669258,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":43324,"kb_used_data":15532,"kb_used_omap":0,"kb_used_meta":27776,"kb_avail":94328516,"statfs":{"total":96636764160,"available":96592400384,"internally_reserved":0,"allocated":15904768,"data_stored":36318126,"data_compressed":1817450,"data_compressed_allocated":14508032,"data_compressed_original":35741696,"omap_allocated":0,"internal_metadata":28442624},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29962,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29962,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":14860288,"data_stored":35954688,"data_compressed":1828096,"data_compressed_allocated":14860288,"data_compressed_original":35954688,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":15581184,"data_stored":31162368,"data_compressed":1608222,"data_compressed_allocated":15581184,"data_compressed_original":31162368,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":21291008,"data_stored":42614784,"data_compressed":2199332,"data_compressed_allocated":21291008,"data_compressed_original":42614784,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12460032,"data_stored":24969216,"data_compressed":1288402,"data_compressed_allocated":12460032,"data_compressed_original":24969216,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-22T03:22:42.477 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-22T03:22:42.660 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:22:42.652+0000 7f82220ce640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:22:42.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:22:42.664+0000 7f82220ce640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:22:42.971 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-22T03:22:42.972 INFO:teuthology.orchestra.run.smithi153.stderr:dumped all 2024-09-22T03:22:42.990 INFO:teuthology.orchestra.run.smithi153.stdout:{"pg_ready":true,"pg_map":{"version":601,"stamp":"2024-09-22T03:22:42.293468+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":44950067,"num_objects":682,"num_object_clones":0,"num_object_copies":2046,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":5554,"num_write_kb":45216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":112,"num_bytes_recovered":3170304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5504,"ondisk_log_size":5504,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":188328,"kb_used_data":67112,"kb_used_omap":0,"kb_used_meta":121152,"kb_avail":377299032,"statfs":{"total":386547056640,"available":386354208768,"internally_reserved":0,"allocated":68722688,"data_stored":136327980,"data_compressed":6890422,"data_compressed_allocated":63119360,"data_compressed_original":134029312,"omap_allocated":0,"internal_metadata":124059648},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":10338304,"num_objects":158,"num_object_clones":0,"num_object_copies":474,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1262,"num_write_kb":10096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000951"},"pg_stats":[{"pgid":"10.7","version":"165'367","reported_seq":327,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-22T03:22:39.330521+0000","last_change":"2024-09-22T03:22:30.549286+0000","last_active":"2024-09-22T03:22:39.330521+0000","last_peered":"2024-09-22T03:22:39.330521+0000","last_clean":"2024-09-22T03:22:39.330521+0000","last_became_active":"2024-09-22T03:22:30.085182+0000","last_became_peered":"2024-09-22T03:22:30.085182+0000","last_unstale":"2024-09-22T03:22:39.330521+0000","last_undegraded":"2024-09-22T03:22:39.330521+0000","last_fullsized":"2024-09-22T03:22:39.330521+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:21:11.969997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:21:11.969997+0000","last_clean_scrub_stamp":"2024-09-22T03:21:11.969997+0000","objects_scrubbed":0,"log_size":367,"log_dups_size":0,"ondisk_log_size":367,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3014656,"num_objects":46,"num_object_clones":0,"num_object_copies":138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":368,"num_write_kb":2944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.6","version":"164'351","reported_seq":415,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-22T03:22:37.691439+0000","last_change":"2024-09-22T03:22:30.891241+0000","last_active":"2024-09-22T03:22:37.691439+0000","last_peered":"2024-09-22T03:22:37.691439+0000","last_clean":"2024-09-22T03:22:37.691439+0000","last_became_active":"2024-09-22T03:22:30.100277+0000","last_became_peered":"2024-09-22T03:22:30.100277+0000","last_unstale":"2024-09-22T03:22:37.691439+0000","last_undegraded":"2024-09-22T03:22:37.691439+0000","last_fullsized":"2024-09-22T03:22:37.691439+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:21:11.969997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:21:11.969997+0000","last_clean_scrub_stamp":"2024-09-22T03:21:11.969997+0000","objects_scrubbed":0,"log_size":351,"log_dups_size":0,"ondisk_log_size":351,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2883584,"num_objects":44,"num_object_clones":0,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":352,"num_write_kb":2816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":524288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.5","version":"165'416","reported_seq":457,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-22T03:22:40.025212+0000","last_change":"2024-09-22T03:22:16.387575+0000","last_active":"2024-09-22T03:22:40.025212+0000","last_peered":"2024-09-22T03:22:40.025212+0000","last_clean":"2024-09-22T03:22:40.025212+0000","last_became_active":"2024-09-22T03:22:16.383638+0000","last_became_peered":"2024-09-22T03:22:16.383638+0000","last_unstale":"2024-09-22T03:22:40.025212+0000","last_undegraded":"2024-09-22T03:22:40.025212+0000","last_fullsized":"2024-09-22T03:22:40.025212+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:21:11.969997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:21:11.969997+0000","last_clean_scrub_stamp":"2024-09-22T03:21:11.969997+0000","objects_scrubbed":0,"log_size":416,"log_dups_size":0,"ondisk_log_size":416,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3407872,"num_objects":52,"num_object_clones":0,"num_object_copies":156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":416,"num_write_kb":3328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.4","version":"165'319","reported_seq":365,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-22T03:22:39.050225+0000","last_change":"2024-09-22T03:22:16.388149+0000","last_active":"2024-09-22T03:22:39.050225+0000","last_peered":"2024-09-22T03:22:39.050225+0000","last_clean":"2024-09-22T03:22:39.050225+0000","last_became_active":"2024-09-22T03:22:16.384355+0000","last_became_peered":"2024-09-22T03:22:16.384355+0000","last_unstale":"2024-09-22T03:22:39.050225+0000","last_undegraded":"2024-09-22T03:22:39.050225+0000","last_fullsized":"2024-09-22T03:22:39.050225+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:21:11.969997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:21:11.969997+0000","last_clean_scrub_stamp":"2024-09-22T03:21:11.969997+0000","objects_scrubbed":0,"log_size":319,"log_dups_size":0,"ondisk_log_size":319,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2621440,"num_objects":40,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":320,"num_write_kb":2560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.3","version":"165'351","reported_seq":339,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-22T03:22:39.240776+0000","last_change":"2024-09-22T03:22:31.459047+0000","last_active":"2024-09-22T03:22:39.240776+0000","last_peered":"2024-09-22T03:22:39.240776+0000","last_clean":"2024-09-22T03:22:39.240776+0000","last_became_active":"2024-09-22T03:22:30.079088+0000","last_became_peered":"2024-09-22T03:22:30.079088+0000","last_unstale":"2024-09-22T03:22:39.240776+0000","last_undegraded":"2024-09-22T03:22:39.240776+0000","last_fullsized":"2024-09-22T03:22:39.240776+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:21:11.969997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:21:11.969997+0000","last_clean_scrub_stamp":"2024-09-22T03:21:11.969997+0000","objects_scrubbed":0,"log_size":351,"log_dups_size":0,"ondisk_log_size":351,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2883584,"num_objects":44,"num_object_clones":0,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":352,"num_write_kb":2816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":8192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.2","version":"164'320","reported_seq":383,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-22T03:22:39.012701+0000","last_change":"2024-09-22T03:22:30.332579+0000","last_active":"2024-09-22T03:22:39.012701+0000","last_peered":"2024-09-22T03:22:39.012701+0000","last_clean":"2024-09-22T03:22:39.012701+0000","last_became_active":"2024-09-22T03:22:30.110457+0000","last_became_peered":"2024-09-22T03:22:30.110457+0000","last_unstale":"2024-09-22T03:22:39.012701+0000","last_undegraded":"2024-09-22T03:22:39.012701+0000","last_fullsized":"2024-09-22T03:22:39.012701+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:21:11.969997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:21:11.969997+0000","last_clean_scrub_stamp":"2024-09-22T03:21:11.969997+0000","objects_scrubbed":0,"log_size":320,"log_dups_size":0,"ondisk_log_size":320,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2621440,"num_objects":40,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":320,"num_write_kb":2560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":196608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1","version":"165'367","reported_seq":336,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-22T03:22:39.175565+0000","last_change":"2024-09-22T03:22:30.754141+0000","last_active":"2024-09-22T03:22:39.175565+0000","last_peered":"2024-09-22T03:22:39.175565+0000","last_clean":"2024-09-22T03:22:39.175565+0000","last_became_active":"2024-09-22T03:22:30.109611+0000","last_became_peered":"2024-09-22T03:22:30.109611+0000","last_unstale":"2024-09-22T03:22:39.175565+0000","last_undegraded":"2024-09-22T03:22:39.175565+0000","last_fullsized":"2024-09-22T03:22:39.175565+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:21:11.969997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:21:11.969997+0000","last_clean_scrub_stamp":"2024-09-22T03:21:11.969997+0000","objects_scrubbed":0,"log_size":367,"log_dups_size":0,"ondisk_log_size":367,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3014656,"num_objects":46,"num_object_clones":0,"num_object_copies":138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":368,"num_write_kb":2944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.0","version":"164'311","reported_seq":379,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-22T03:22:37.159536+0000","last_change":"2024-09-22T03:22:31.226107+0000","last_active":"2024-09-22T03:22:37.159536+0000","last_peered":"2024-09-22T03:22:37.159536+0000","last_clean":"2024-09-22T03:22:37.159536+0000","last_became_active":"2024-09-22T03:22:30.105137+0000","last_became_peered":"2024-09-22T03:22:30.105137+0000","last_unstale":"2024-09-22T03:22:37.159536+0000","last_undegraded":"2024-09-22T03:22:37.159536+0000","last_fullsized":"2024-09-22T03:22:37.159536+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:21:11.969997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:21:11.969997+0000","last_clean_scrub_stamp":"2024-09-22T03:21:11.969997+0000","objects_scrubbed":0,"log_size":311,"log_dups_size":0,"ondisk_log_size":311,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2555904,"num_objects":39,"num_object_clones":0,"num_object_copies":117,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":312,"num_write_kb":2496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":655360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.b","version":"164'347","reported_seq":402,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-22T03:22:37.705464+0000","last_change":"2024-09-22T03:22:16.420237+0000","last_active":"2024-09-22T03:22:37.705464+0000","last_peered":"2024-09-22T03:22:37.705464+0000","last_clean":"2024-09-22T03:22:37.705464+0000","last_became_active":"2024-09-22T03:22:16.396604+0000","last_became_peered":"2024-09-22T03:22:16.396604+0000","last_unstale":"2024-09-22T03:22:37.705464+0000","last_undegraded":"2024-09-22T03:22:37.705464+0000","last_fullsized":"2024-09-22T03:22:37.705464+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:21:11.969997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:21:11.969997+0000","last_clean_scrub_stamp":"2024-09-22T03:21:11.969997+0000","objects_scrubbed":0,"log_size":347,"log_dups_size":0,"ondisk_log_size":347,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2850816,"num_objects":44,"num_object_clones":0,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":348,"num_write_kb":2784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":233,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-22T03:22:39.013931+0000","last_change":"2024-09-22T03:22:39.015859+0000","last_active":"2024-09-22T03:22:39.013931+0000","last_peered":"2024-09-22T03:22:39.013931+0000","last_clean":"2024-09-22T03:22:39.013931+0000","last_became_active":"2024-09-22T03:22:30.096861+0000","last_became_peered":"2024-09-22T03:22:30.096861+0000","last_unstale":"2024-09-22T03:22:39.013931+0000","last_undegraded":"2024-09-22T03:22:39.013931+0000","last_fullsized":"2024-09-22T03:22:39.013931+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.a","version":"165'343","reported_seq":330,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-22T03:22:39.292396+0000","last_change":"2024-09-22T03:22:31.215850+0000","last_active":"2024-09-22T03:22:39.292396+0000","last_peered":"2024-09-22T03:22:39.292396+0000","last_clean":"2024-09-22T03:22:39.292396+0000","last_became_active":"2024-09-22T03:22:30.129087+0000","last_became_peered":"2024-09-22T03:22:30.129087+0000","last_unstale":"2024-09-22T03:22:39.292396+0000","last_undegraded":"2024-09-22T03:22:39.292396+0000","last_fullsized":"2024-09-22T03:22:39.292396+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:21:11.969997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:21:11.969997+0000","last_clean_scrub_stamp":"2024-09-22T03:21:11.969997+0000","objects_scrubbed":0,"log_size":343,"log_dups_size":0,"ondisk_log_size":343,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2818048,"num_objects":43,"num_object_clones":0,"num_object_copies":129,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":344,"num_write_kb":2752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":366,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-22T03:22:39.022294+0000","last_change":"2024-09-22T03:22:39.030882+0000","last_active":"2024-09-22T03:22:39.022294+0000","last_peered":"2024-09-22T03:22:39.022294+0000","last_clean":"2024-09-22T03:22:39.022294+0000","last_became_active":"2024-09-22T03:22:30.117777+0000","last_became_peered":"2024-09-22T03:22:30.117777+0000","last_unstale":"2024-09-22T03:22:39.022294+0000","last_undegraded":"2024-09-22T03:22:39.022294+0000","last_fullsized":"2024-09-22T03:22:39.022294+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.9","version":"164'304","reported_seq":379,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-22T03:22:39.012343+0000","last_change":"2024-09-22T03:22:30.438260+0000","last_active":"2024-09-22T03:22:39.012343+0000","last_peered":"2024-09-22T03:22:39.012343+0000","last_clean":"2024-09-22T03:22:39.012343+0000","last_became_active":"2024-09-22T03:22:30.100388+0000","last_became_peered":"2024-09-22T03:22:30.100388+0000","last_unstale":"2024-09-22T03:22:39.012343+0000","last_undegraded":"2024-09-22T03:22:39.012343+0000","last_fullsized":"2024-09-22T03:22:39.012343+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:21:11.969997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:21:11.969997+0000","last_clean_scrub_stamp":"2024-09-22T03:21:11.969997+0000","objects_scrubbed":0,"log_size":304,"log_dups_size":0,"ondisk_log_size":304,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2490368,"num_objects":38,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":304,"num_write_kb":2432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":589824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":397,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-22T03:22:40.027260+0000","last_change":"2024-09-22T03:22:25.014526+0000","last_active":"2024-09-22T03:22:40.027260+0000","last_peered":"2024-09-22T03:22:40.027260+0000","last_clean":"2024-09-22T03:22:40.027260+0000","last_became_active":"2024-09-22T03:22:16.387817+0000","last_became_peered":"2024-09-22T03:22:16.387817+0000","last_unstale":"2024-09-22T03:22:40.027260+0000","last_undegraded":"2024-09-22T03:22:40.027260+0000","last_fullsized":"2024-09-22T03:22:40.027260+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.8","version":"166'303","reported_seq":353,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-22T03:22:40.332304+0000","last_change":"2024-09-22T03:22:30.185137+0000","last_active":"2024-09-22T03:22:40.332304+0000","last_peered":"2024-09-22T03:22:40.332304+0000","last_clean":"2024-09-22T03:22:40.332304+0000","last_became_active":"2024-09-22T03:22:30.053731+0000","last_became_peered":"2024-09-22T03:22:30.053731+0000","last_unstale":"2024-09-22T03:22:40.332304+0000","last_undegraded":"2024-09-22T03:22:40.332304+0000","last_fullsized":"2024-09-22T03:22:40.332304+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:21:11.969997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:21:11.969997+0000","last_clean_scrub_stamp":"2024-09-22T03:21:11.969997+0000","objects_scrubbed":0,"log_size":303,"log_dups_size":0,"ondisk_log_size":303,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2490368,"num_objects":38,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":304,"num_write_kb":2432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":281,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-22T03:22:39.012682+0000","last_change":"2024-09-22T03:22:39.013700+0000","last_active":"2024-09-22T03:22:39.012682+0000","last_peered":"2024-09-22T03:22:39.012682+0000","last_clean":"2024-09-22T03:22:39.012682+0000","last_became_active":"2024-09-22T03:22:30.068029+0000","last_became_peered":"2024-09-22T03:22:30.068029+0000","last_unstale":"2024-09-22T03:22:39.012682+0000","last_undegraded":"2024-09-22T03:22:39.012682+0000","last_fullsized":"2024-09-22T03:22:39.012682+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"12'97","reported_seq":486,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-22T03:22:39.024706+0000","last_change":"2024-09-22T03:22:30.096425+0000","last_active":"2024-09-22T03:22:39.024706+0000","last_peered":"2024-09-22T03:22:39.024706+0000","last_clean":"2024-09-22T03:22:39.024706+0000","last_became_active":"2024-09-22T03:22:30.088734+0000","last_became_peered":"2024-09-22T03:22:30.088734+0000","last_unstale":"2024-09-22T03:22:39.024706+0000","last_undegraded":"2024-09-22T03:22:39.024706+0000","last_fullsized":"2024-09-22T03:22:39.024706+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.f","version":"164'311","reported_seq":383,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-22T03:22:37.638721+0000","last_change":"2024-09-22T03:22:30.515318+0000","last_active":"2024-09-22T03:22:37.638721+0000","last_peered":"2024-09-22T03:22:37.638721+0000","last_clean":"2024-09-22T03:22:37.638721+0000","last_became_active":"2024-09-22T03:22:30.054165+0000","last_became_peered":"2024-09-22T03:22:30.054165+0000","last_unstale":"2024-09-22T03:22:37.638721+0000","last_undegraded":"2024-09-22T03:22:37.638721+0000","last_fullsized":"2024-09-22T03:22:37.638721+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:21:11.969997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:21:11.969997+0000","last_clean_scrub_stamp":"2024-09-22T03:21:11.969997+0000","objects_scrubbed":0,"log_size":311,"log_dups_size":0,"ondisk_log_size":311,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2555904,"num_objects":39,"num_object_clones":0,"num_object_copies":117,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":312,"num_write_kb":2496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":786432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":322,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-22T03:22:30.106450+0000","last_change":"2024-09-22T03:22:30.106962+0000","last_active":"2024-09-22T03:22:30.106450+0000","last_peered":"2024-09-22T03:22:30.106450+0000","last_clean":"2024-09-22T03:21:50.022603+0000","last_became_active":"2024-09-22T03:22:30.102934+0000","last_became_peered":"2024-09-22T03:22:30.102934+0000","last_unstale":"2024-09-22T03:22:30.106450+0000","last_undegraded":"2024-09-22T03:22:30.106450+0000","last_fullsized":"2024-09-22T03:22:30.106450+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.e","version":"165'346","reported_seq":327,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-22T03:22:39.367480+0000","last_change":"2024-09-22T03:22:31.235882+0000","last_active":"2024-09-22T03:22:39.367480+0000","last_peered":"2024-09-22T03:22:39.367480+0000","last_clean":"2024-09-22T03:22:39.367480+0000","last_became_active":"2024-09-22T03:22:30.135157+0000","last_became_peered":"2024-09-22T03:22:30.135157+0000","last_unstale":"2024-09-22T03:22:39.367480+0000","last_undegraded":"2024-09-22T03:22:39.367480+0000","last_fullsized":"2024-09-22T03:22:39.367480+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:21:11.969997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:21:11.969997+0000","last_clean_scrub_stamp":"2024-09-22T03:21:11.969997+0000","objects_scrubbed":0,"log_size":346,"log_dups_size":0,"ondisk_log_size":346,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2842624,"num_objects":44,"num_object_clones":0,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":347,"num_write_kb":2776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":367,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-22T03:22:40.026301+0000","last_change":"2024-09-22T03:22:39.026321+0000","last_active":"2024-09-22T03:22:40.026301+0000","last_peered":"2024-09-22T03:22:40.026301+0000","last_clean":"2024-09-22T03:22:40.026301+0000","last_became_active":"2024-09-22T03:22:30.055276+0000","last_became_peered":"2024-09-22T03:22:30.055276+0000","last_unstale":"2024-09-22T03:22:40.026301+0000","last_undegraded":"2024-09-22T03:22:40.026301+0000","last_fullsized":"2024-09-22T03:22:40.026301+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.d","version":"164'312","reported_seq":292,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-22T03:22:39.031852+0000","last_change":"2024-09-22T03:22:30.910625+0000","last_active":"2024-09-22T03:22:39.031852+0000","last_peered":"2024-09-22T03:22:39.031852+0000","last_clean":"2024-09-22T03:22:39.031852+0000","last_became_active":"2024-09-22T03:22:30.111068+0000","last_became_peered":"2024-09-22T03:22:30.111068+0000","last_unstale":"2024-09-22T03:22:39.031852+0000","last_undegraded":"2024-09-22T03:22:39.031852+0000","last_fullsized":"2024-09-22T03:22:39.031852+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:21:11.969997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:21:11.969997+0000","last_clean_scrub_stamp":"2024-09-22T03:21:11.969997+0000","objects_scrubbed":0,"log_size":312,"log_dups_size":0,"ondisk_log_size":312,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2555904,"num_objects":39,"num_object_clones":0,"num_object_copies":117,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":312,"num_write_kb":2496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":400,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-22T03:22:30.012451+0000","last_change":"2024-09-22T03:22:25.010645+0000","last_active":"2024-09-22T03:22:30.012451+0000","last_peered":"2024-09-22T03:22:30.012451+0000","last_clean":"2024-09-22T03:22:30.012451+0000","last_became_active":"2024-09-22T03:22:16.384462+0000","last_became_peered":"2024-09-22T03:22:16.384462+0000","last_unstale":"2024-09-22T03:22:30.012451+0000","last_undegraded":"2024-09-22T03:22:30.012451+0000","last_fullsized":"2024-09-22T03:22:30.012451+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.c","version":"164'336","reported_seq":316,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-22T03:22:39.027978+0000","last_change":"2024-09-22T03:22:30.382901+0000","last_active":"2024-09-22T03:22:39.027978+0000","last_peered":"2024-09-22T03:22:39.027978+0000","last_clean":"2024-09-22T03:22:39.027978+0000","last_became_active":"2024-09-22T03:22:30.094983+0000","last_became_peered":"2024-09-22T03:22:30.094983+0000","last_unstale":"2024-09-22T03:22:39.027978+0000","last_undegraded":"2024-09-22T03:22:39.027978+0000","last_fullsized":"2024-09-22T03:22:39.027978+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:21:11.969997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:21:11.969997+0000","last_clean_scrub_stamp":"2024-09-22T03:21:11.969997+0000","objects_scrubbed":0,"log_size":336,"log_dups_size":0,"ondisk_log_size":336,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2752512,"num_objects":42,"num_object_clones":0,"num_object_copies":126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":336,"num_write_kb":2688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":16384,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":407,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-09-22T03:22:30.107638+0000","last_change":"2024-09-22T03:22:30.108468+0000","last_active":"2024-09-22T03:22:30.107638+0000","last_peered":"2024-09-22T03:22:30.107638+0000","last_clean":"2024-09-22T03:22:16.362770+0000","last_became_active":"2024-09-22T03:22:30.101199+0000","last_became_peered":"2024-09-22T03:22:30.101199+0000","last_unstale":"2024-09-22T03:22:30.107638+0000","last_undegraded":"2024-09-22T03:22:30.107638+0000","last_fullsized":"2024-09-22T03:22:30.107638+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":16,"stat_sum":{"num_bytes":44359680,"num_objects":678,"num_object_clones":0,"num_object_copies":2034,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5415,"num_write_kb":43320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":112,"num_bytes_recovered":3170304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":64192512,"data_stored":134701056,"data_compressed":6924052,"data_compressed_allocated":64192512,"data_compressed_original":134701056,"omap_allocated":0,"internal_metadata":0},"log_size":5404,"ondisk_log_size":5404,"up":48,"acting":48,"num_store_stats":4},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1180736,"data_compressed":59924,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":155,"seq":665719930930,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":44244,"kb_used_data":13508,"kb_used_omap":0,"kb_used_meta":30720,"kb_avail":94327596,"statfs":{"total":96636764160,"available":96591458304,"internally_reserved":0,"allocated":13832192,"data_stored":25499242,"data_compressed":1286710,"data_compressed_allocated":12443648,"data_compressed_original":24936448,"omap_allocated":0,"internal_metadata":31457280},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":159,"seq":682899800089,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":55400,"kb_used_data":22104,"kb_used_omap":0,"kb_used_meta":33280,"kb_avail":94316440,"statfs":{"total":96636764160,"available":96580034560,"internally_reserved":0,"allocated":22634496,"data_stored":43541843,"data_compressed":2217450,"data_compressed_allocated":21217280,"data_compressed_original":42958848,"omap_allocated":0,"internal_metadata":34078720},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":138,"seq":592705487037,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":45360,"kb_used_data":15968,"kb_used_omap":0,"kb_used_meta":29376,"kb_avail":94326480,"statfs":{"total":96636764160,"available":96590315520,"internally_reserved":0,"allocated":16351232,"data_stored":30968769,"data_compressed":1568812,"data_compressed_allocated":14950400,"data_compressed_original":30392320,"omap_allocated":0,"internal_metadata":30081024},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":163,"seq":700079669258,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":43324,"kb_used_data":15532,"kb_used_omap":0,"kb_used_meta":27776,"kb_avail":94328516,"statfs":{"total":96636764160,"available":96592400384,"internally_reserved":0,"allocated":15904768,"data_stored":36318126,"data_compressed":1817450,"data_compressed_allocated":14508032,"data_compressed_original":35741696,"omap_allocated":0,"internal_metadata":28442624},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29962,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29962,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":14860288,"data_stored":35954688,"data_compressed":1828096,"data_compressed_allocated":14860288,"data_compressed_original":35954688,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":15581184,"data_stored":31162368,"data_compressed":1608222,"data_compressed_allocated":15581184,"data_compressed_original":31162368,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":21291008,"data_stored":42614784,"data_compressed":2199332,"data_compressed_allocated":21291008,"data_compressed_original":42614784,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12460032,"data_stored":24969216,"data_compressed":1288402,"data_compressed_allocated":12460032,"data_compressed_original":24969216,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-22T03:22:42.992 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-09-22T03:22:42.992 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-22T03:22:42.992 INFO:tasks.thrashosds.thrasher:Killing osd 1, live_osds are [1, 3, 2, 0] 2024-09-22T03:22:42.992 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-09-22T03:22:42.992 INFO:teuthology.orchestra.run:waiting for 300 2024-09-22T03:22:43.076 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 89 16 6727 6711 0.589028 2.58594 0.00994796 0.21205 2024-09-22T03:22:43.132 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-22T03:22:43.233 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:22:43.230+0000 7f627d8d7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:22:43.236 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:22:43.234+0000 7f627d8d7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:22:44.076 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 90 16 6730 6714 0.582744 0.0234375 0.00980611 0.21196 2024-09-22T03:22:45.076 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 91 16 6730 6714 0.57634 0 - 0.21196 2024-09-22T03:22:46.076 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 92 16 6730 6714 0.570075 0 - 0.21196 2024-09-22T03:22:47.076 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 93 16 6730 6714 0.563945 0 - 0.21196 2024-09-22T03:22:48.077 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 94 16 6730 6714 0.557946 0 - 0.21196 2024-09-22T03:22:48.093 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~0s 2024-09-22T03:22:49.077 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 95 16 6730 6714 0.552072 0 - 0.21196 2024-09-22T03:22:49.094 INFO:tasks.ceph.osd.1:Stopped 2024-09-22T03:22:50.077 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 96 16 6730 6714 0.546321 0 - 0.21196 2024-09-22T03:22:51.077 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 97 16 6730 6714 0.540689 0 - 0.21196 2024-09-22T03:22:52.077 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 98 16 6730 6714 0.535171 0 - 0.21196 2024-09-22T03:22:53.077 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 99 16 6730 6714 0.529766 0 - 0.21196 2024-09-22T03:22:53.597 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 has been restored 2024-09-22T03:22:54.077 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:2024-09-22T03:22:54.077133+0000 min lat: 0.00759976 max lat: 68.3554 avg lat: 0.21196 2024-09-22T03:22:54.078 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-22T03:22:54.078 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 100 16 6730 6714 0.524468 0 - 0.21196 2024-09-22T03:22:54.095 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [1] live_osds: [3, 2, 0] 2024-09-22T03:22:54.095 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-22T03:22:54.096 INFO:tasks.thrashosds.thrasher:Reviving osd 1 2024-09-22T03:22:54.096 INFO:tasks.ceph.osd.1:Restarting daemon 2024-09-22T03:22:54.096 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2>> /var/log/ceph/ceph-osd.1.log 2024-09-22T03:22:54.099 INFO:tasks.ceph.osd.1:Started 2024-09-22T03:22:54.099 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-22T03:22:54.209 INFO:teuthology.orchestra.run.smithi153.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-09-22T03:22:54.213 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-22T03:22:54.213 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2024-09-22T03:22:55.078 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 101 16 6730 6714 0.519275 0 - 0.21196 2024-09-22T03:22:56.078 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 102 16 6730 6714 0.514184 0 - 0.21196 2024-09-22T03:22:57.078 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 103 16 6730 6714 0.509192 0 - 0.21196 2024-09-22T03:22:58.078 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 104 16 6730 6714 0.504296 0 - 0.21196 2024-09-22T03:22:59.078 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 105 16 6730 6714 0.499494 0 - 0.21196 2024-09-22T03:22:59.214 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-22T03:22:59.311 INFO:teuthology.orchestra.run.smithi153.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-09-22T03:22:59.314 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-22T03:22:59.314 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2024-09-22T03:23:00.078 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 106 16 6730 6714 0.494781 0 - 0.21196 2024-09-22T03:23:01.078 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 107 16 6730 6714 0.490157 0 - 0.21196 2024-09-22T03:23:02.078 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 108 16 6730 6714 0.485619 0 - 0.21196 2024-09-22T03:23:03.078 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 109 16 6730 6714 0.481164 0 - 0.21196 2024-09-22T03:23:03.982 INFO:teuthology.orchestra.run.smithi153.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-09-22T03:23:04.078 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 110 16 6730 6714 0.47679 0 - 0.21196 2024-09-22T03:23:04.081 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-22T03:23:04.081 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-22T03:23:04.315 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-22T03:23:04.416 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-22T03:23:04.416 INFO:teuthology.orchestra.run.smithi153.stdout: "ops_in_flight": [] 2024-09-22T03:23:04.416 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-22T03:23:05.078 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 111 16 6730 6714 0.472495 0 - 0.21196 2024-09-22T03:23:06.079 INFO:tasks.radosbench.radosbench.0.smithi153.stdout: 112 16 6730 6714 0.468276 0 - 0.21196 2024-09-22T03:23:06.082 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-22T03:23:06.208 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:23:06.205+0000 7f70dbad7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:23:06.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:23:06.210+0000 7f70dbad7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:23:07.079 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:Total time run: 112.154 2024-09-22T03:23:07.079 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:Total writes made: 6730 2024-09-22T03:23:07.079 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:Write size: 8192 2024-09-22T03:23:07.079 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:Object size: 65536 2024-09-22T03:23:07.079 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:Bandwidth (MB/sec): 0.468805 2024-09-22T03:23:07.079 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:Stddev Bandwidth: 0.972605 2024-09-22T03:23:07.079 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:Max bandwidth (MB/sec): 2.97656 2024-09-22T03:23:07.079 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:Min bandwidth (MB/sec): 0 2024-09-22T03:23:07.080 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:Average IOPS: 60 2024-09-22T03:23:07.080 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:Stddev IOPS: 124.498 2024-09-22T03:23:07.080 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:Max IOPS: 381 2024-09-22T03:23:07.080 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:Min IOPS: 0 2024-09-22T03:23:07.080 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:Average Latency(s): 0.266542 2024-09-22T03:23:07.080 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:Stddev Latency(s): 3.10294 2024-09-22T03:23:07.080 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:Max latency(s): 68.3554 2024-09-22T03:23:07.080 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:Min latency(s): 0.00759976 2024-09-22T03:23:07.094 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:Cleaning up (deleting benchmark objects) 2024-09-22T03:23:08.280 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-22T03:23:08.423 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:23:08.419+0000 7f5b07397640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:23:08.427 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:23:08.426+0000 7f5b07397640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:23:09.376 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:Removed 842 objects 2024-09-22T03:23:09.376 INFO:tasks.radosbench.radosbench.0.smithi153.stdout:Clean up completed and total clean up time :2.2816 2024-09-22T03:23:09.425 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [3, 2, 0, 1] 2024-09-22T03:23:09.425 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-22T03:23:09.425 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-22T03:23:09.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:23:09.527+0000 7f751f2d7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:23:09.535 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:23:09.534+0000 7f751f2d7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:23:09.711 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-22T03:23:09.712 INFO:teuthology.orchestra.run.smithi153.stderr:dumped all 2024-09-22T03:23:09.723 INFO:teuthology.orchestra.run.smithi153.stdout:{"pg_ready":true,"pg_map":{"version":618,"stamp":"2024-09-22T03:23:08.974345+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":55665203,"num_objects":846,"num_object_clones":0,"num_object_copies":2538,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":6862,"num_write_kb":55680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":112,"num_bytes_recovered":3170304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6822,"ondisk_log_size":6822,"up":59,"acting":59,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":238212,"kb_used_data":82308,"kb_used_omap":0,"kb_used_meta":155840,"kb_avail":377249148,"statfs":{"total":386547056640,"available":386303127552,"internally_reserved":0,"allocated":84283392,"data_stored":167228351,"data_compressed":8483017,"data_compressed_allocated":78540800,"data_compressed_original":164872192,"omap_allocated":0,"internal_metadata":159580160},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":131072,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.994838"},"pg_stats":[{"pgid":"10.7","version":"168'432","reported_seq":403,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-22T03:23:04.015383+0000","last_change":"2024-09-22T03:22:30.549286+0000","last_active":"2024-09-22T03:23:04.015383+0000","last_peered":"2024-09-22T03:23:04.015383+0000","last_clean":"2024-09-22T03:23:04.015383+0000","last_became_active":"2024-09-22T03:22:30.085182+0000","last_became_peered":"2024-09-22T03:22:30.085182+0000","last_unstale":"2024-09-22T03:23:04.015383+0000","last_undegraded":"2024-09-22T03:23:04.015383+0000","last_fullsized":"2024-09-22T03:23:04.015383+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:21:11.969997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:21:11.969997+0000","last_clean_scrub_stamp":"2024-09-22T03:21:11.969997+0000","objects_scrubbed":0,"log_size":432,"log_dups_size":0,"ondisk_log_size":432,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3538944,"num_objects":54,"num_object_clones":0,"num_object_copies":162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":432,"num_write_kb":3456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.6","version":"168'464","reported_seq":540,"reported_epoch":169,"state":"peering","last_fresh":"2024-09-22T03:22:42.773202+0000","last_change":"2024-09-22T03:23:04.007610+0000","last_active":"2024-09-22T03:23:04.007610+0000","last_peered":"2024-09-22T03:22:42.773202+0000","last_clean":"2024-09-22T03:22:42.773202+0000","last_became_active":"2024-09-22T03:22:30.100277+0000","last_became_peered":"2024-09-22T03:22:30.100277+0000","last_unstale":"2024-09-22T03:22:42.773202+0000","last_undegraded":"2024-09-22T03:22:42.773202+0000","last_fullsized":"2024-09-22T03:22:42.773202+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:21:11.969997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:21:11.969997+0000","last_clean_scrub_stamp":"2024-09-22T03:21:11.969997+0000","objects_scrubbed":0,"log_size":464,"log_dups_size":0,"ondisk_log_size":464,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3801088,"num_objects":58,"num_object_clones":0,"num_object_copies":174,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":464,"num_write_kb":3712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":524288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.5","version":"168'480","reported_seq":529,"reported_epoch":170,"state":"peering","last_fresh":"2024-09-22T03:23:03.994805+0000","last_change":"2024-09-22T03:23:03.994805+0000","last_active":"2024-09-22T03:23:03.994805+0000","last_peered":"2024-09-22T03:22:42.167317+0000","last_clean":"2024-09-22T03:22:42.167317+0000","last_became_active":"2024-09-22T03:22:16.383638+0000","last_became_peered":"2024-09-22T03:22:16.383638+0000","last_unstale":"2024-09-22T03:23:03.994805+0000","last_undegraded":"2024-09-22T03:23:03.994805+0000","last_fullsized":"2024-09-22T03:23:03.994805+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:21:11.969997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:21:11.969997+0000","last_clean_scrub_stamp":"2024-09-22T03:21:11.969997+0000","objects_scrubbed":0,"log_size":480,"log_dups_size":0,"ondisk_log_size":480,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3932160,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":480,"num_write_kb":3840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,3],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.4","version":"168'424","reported_seq":479,"reported_epoch":169,"state":"peering","last_fresh":"2024-09-22T03:22:42.980936+0000","last_change":"2024-09-22T03:23:04.003973+0000","last_active":"2024-09-22T03:23:04.003973+0000","last_peered":"2024-09-22T03:22:42.980936+0000","last_clean":"2024-09-22T03:22:42.980936+0000","last_became_active":"2024-09-22T03:22:16.384355+0000","last_became_peered":"2024-09-22T03:22:16.384355+0000","last_unstale":"2024-09-22T03:22:42.980936+0000","last_undegraded":"2024-09-22T03:22:42.980936+0000","last_fullsized":"2024-09-22T03:22:42.980936+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:21:11.969997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:21:11.969997+0000","last_clean_scrub_stamp":"2024-09-22T03:21:11.969997+0000","objects_scrubbed":0,"log_size":424,"log_dups_size":0,"ondisk_log_size":424,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3473408,"num_objects":53,"num_object_clones":0,"num_object_copies":159,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":424,"num_write_kb":3392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.3","version":"168'432","reported_seq":431,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-22T03:23:03.995117+0000","last_change":"2024-09-22T03:22:31.459047+0000","last_active":"2024-09-22T03:23:03.995117+0000","last_peered":"2024-09-22T03:23:03.995117+0000","last_clean":"2024-09-22T03:23:03.995117+0000","last_became_active":"2024-09-22T03:22:30.079088+0000","last_became_peered":"2024-09-22T03:22:30.079088+0000","last_unstale":"2024-09-22T03:23:03.995117+0000","last_undegraded":"2024-09-22T03:23:03.995117+0000","last_fullsized":"2024-09-22T03:23:03.995117+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:21:11.969997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:21:11.969997+0000","last_clean_scrub_stamp":"2024-09-22T03:21:11.969997+0000","objects_scrubbed":0,"log_size":432,"log_dups_size":0,"ondisk_log_size":432,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3538944,"num_objects":54,"num_object_clones":0,"num_object_copies":162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":432,"num_write_kb":3456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":8192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.2","version":"168'424","reported_seq":497,"reported_epoch":169,"state":"peering","last_fresh":"2024-09-22T03:22:42.793676+0000","last_change":"2024-09-22T03:23:03.998471+0000","last_active":"2024-09-22T03:23:03.998471+0000","last_peered":"2024-09-22T03:22:42.793676+0000","last_clean":"2024-09-22T03:22:42.793676+0000","last_became_active":"2024-09-22T03:22:30.110457+0000","last_became_peered":"2024-09-22T03:22:30.110457+0000","last_unstale":"2024-09-22T03:22:42.793676+0000","last_undegraded":"2024-09-22T03:22:42.793676+0000","last_fullsized":"2024-09-22T03:22:42.793676+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:21:11.969997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:21:11.969997+0000","last_clean_scrub_stamp":"2024-09-22T03:21:11.969997+0000","objects_scrubbed":0,"log_size":424,"log_dups_size":0,"ondisk_log_size":424,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3473408,"num_objects":53,"num_object_clones":0,"num_object_copies":159,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":424,"num_write_kb":3392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":196608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1","version":"168'440","reported_seq":418,"reported_epoch":169,"state":"peering","last_fresh":"2024-09-22T03:22:42.380263+0000","last_change":"2024-09-22T03:23:04.006901+0000","last_active":"2024-09-22T03:23:04.006901+0000","last_peered":"2024-09-22T03:22:42.380263+0000","last_clean":"2024-09-22T03:22:42.380263+0000","last_became_active":"2024-09-22T03:22:30.109611+0000","last_became_peered":"2024-09-22T03:22:30.109611+0000","last_unstale":"2024-09-22T03:22:42.380263+0000","last_undegraded":"2024-09-22T03:22:42.380263+0000","last_fullsized":"2024-09-22T03:22:42.380263+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:21:11.969997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:21:11.969997+0000","last_clean_scrub_stamp":"2024-09-22T03:21:11.969997+0000","objects_scrubbed":0,"log_size":440,"log_dups_size":0,"ondisk_log_size":440,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3604480,"num_objects":55,"num_object_clones":0,"num_object_copies":165,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":440,"num_write_kb":3520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.0","version":"168'423","reported_seq":503,"reported_epoch":168,"state":"stale+active+clean","last_fresh":"2024-09-22T03:22:42.336064+0000","last_change":"2024-09-22T03:22:31.226107+0000","last_active":"2024-09-22T03:22:42.336064+0000","last_peered":"2024-09-22T03:22:42.336064+0000","last_clean":"2024-09-22T03:22:42.336064+0000","last_became_active":"2024-09-22T03:22:30.105137+0000","last_became_peered":"2024-09-22T03:22:30.105137+0000","last_unstale":"2024-09-22T03:23:03.979529+0000","last_undegraded":"2024-09-22T03:22:42.336064+0000","last_fullsized":"2024-09-22T03:22:42.336064+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:21:11.969997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:21:11.969997+0000","last_clean_scrub_stamp":"2024-09-22T03:21:11.969997+0000","objects_scrubbed":0,"log_size":423,"log_dups_size":0,"ondisk_log_size":423,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3473408,"num_objects":53,"num_object_clones":0,"num_object_copies":159,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":424,"num_write_kb":3392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":655360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.b","version":"168'480","reported_seq":547,"reported_epoch":169,"state":"peering","last_fresh":"2024-09-22T03:22:42.744365+0000","last_change":"2024-09-22T03:23:03.993044+0000","last_active":"2024-09-22T03:23:03.993044+0000","last_peered":"2024-09-22T03:22:42.744365+0000","last_clean":"2024-09-22T03:22:42.744365+0000","last_became_active":"2024-09-22T03:22:16.396604+0000","last_became_peered":"2024-09-22T03:22:16.396604+0000","last_unstale":"2024-09-22T03:22:42.744365+0000","last_undegraded":"2024-09-22T03:22:42.744365+0000","last_fullsized":"2024-09-22T03:22:42.744365+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:21:11.969997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:21:11.969997+0000","last_clean_scrub_stamp":"2024-09-22T03:21:11.969997+0000","objects_scrubbed":0,"log_size":480,"log_dups_size":0,"ondisk_log_size":480,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3932160,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":480,"num_write_kb":3840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":245,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-22T03:23:04.001923+0000","last_change":"2024-09-22T03:22:39.015859+0000","last_active":"2024-09-22T03:23:04.001923+0000","last_peered":"2024-09-22T03:23:04.001923+0000","last_clean":"2024-09-22T03:23:04.001923+0000","last_became_active":"2024-09-22T03:22:30.096861+0000","last_became_peered":"2024-09-22T03:22:30.096861+0000","last_unstale":"2024-09-22T03:23:04.001923+0000","last_undegraded":"2024-09-22T03:23:04.001923+0000","last_fullsized":"2024-09-22T03:23:04.001923+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.a","version":"168'418","reported_seq":414,"reported_epoch":169,"state":"peering","last_fresh":"2024-09-22T03:22:43.093885+0000","last_change":"2024-09-22T03:23:03.996353+0000","last_active":"2024-09-22T03:23:03.996353+0000","last_peered":"2024-09-22T03:22:43.093885+0000","last_clean":"2024-09-22T03:22:43.093885+0000","last_became_active":"2024-09-22T03:22:30.129087+0000","last_became_peered":"2024-09-22T03:22:30.129087+0000","last_unstale":"2024-09-22T03:22:43.093885+0000","last_undegraded":"2024-09-22T03:22:43.093885+0000","last_fullsized":"2024-09-22T03:22:43.093885+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:21:11.969997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:21:11.969997+0000","last_clean_scrub_stamp":"2024-09-22T03:21:11.969997+0000","objects_scrubbed":0,"log_size":418,"log_dups_size":0,"ondisk_log_size":418,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3424256,"num_objects":53,"num_object_clones":0,"num_object_copies":159,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":418,"num_write_kb":3344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":376,"reported_epoch":169,"state":"peering","last_fresh":"2024-09-22T03:22:42.067830+0000","last_change":"2024-09-22T03:23:03.988221+0000","last_active":"2024-09-22T03:23:03.988221+0000","last_peered":"2024-09-22T03:22:42.067830+0000","last_clean":"2024-09-22T03:22:42.067830+0000","last_became_active":"2024-09-22T03:22:30.117777+0000","last_became_peered":"2024-09-22T03:22:30.117777+0000","last_unstale":"2024-09-22T03:22:42.067830+0000","last_undegraded":"2024-09-22T03:22:42.067830+0000","last_fullsized":"2024-09-22T03:22:42.067830+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.9","version":"168'352","reported_seq":437,"reported_epoch":169,"state":"peering","last_fresh":"2024-09-22T03:22:42.129912+0000","last_change":"2024-09-22T03:23:04.007585+0000","last_active":"2024-09-22T03:23:04.007585+0000","last_peered":"2024-09-22T03:22:42.129912+0000","last_clean":"2024-09-22T03:22:42.129912+0000","last_became_active":"2024-09-22T03:22:30.100388+0000","last_became_peered":"2024-09-22T03:22:30.100388+0000","last_unstale":"2024-09-22T03:22:42.129912+0000","last_undegraded":"2024-09-22T03:22:42.129912+0000","last_fullsized":"2024-09-22T03:22:42.129912+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:21:11.969997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:21:11.969997+0000","last_clean_scrub_stamp":"2024-09-22T03:21:11.969997+0000","objects_scrubbed":0,"log_size":352,"log_dups_size":0,"ondisk_log_size":352,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2883584,"num_objects":44,"num_object_clones":0,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":352,"num_write_kb":2816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":589824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":405,"reported_epoch":170,"state":"peering","last_fresh":"2024-09-22T03:23:04.011692+0000","last_change":"2024-09-22T03:23:04.011692+0000","last_active":"2024-09-22T03:23:04.011692+0000","last_peered":"2024-09-22T03:22:42.041758+0000","last_clean":"2024-09-22T03:22:42.041758+0000","last_became_active":"2024-09-22T03:22:16.387817+0000","last_became_peered":"2024-09-22T03:22:16.387817+0000","last_unstale":"2024-09-22T03:23:04.011692+0000","last_undegraded":"2024-09-22T03:23:04.011692+0000","last_fullsized":"2024-09-22T03:23:04.011692+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,3],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.8","version":"168'344","reported_seq":405,"reported_epoch":170,"state":"active+clean","last_fresh":"2024-09-22T03:23:05.009138+0000","last_change":"2024-09-22T03:22:30.185137+0000","last_active":"2024-09-22T03:23:05.009138+0000","last_peered":"2024-09-22T03:23:05.009138+0000","last_clean":"2024-09-22T03:23:05.009138+0000","last_became_active":"2024-09-22T03:22:30.053731+0000","last_became_peered":"2024-09-22T03:22:30.053731+0000","last_unstale":"2024-09-22T03:23:05.009138+0000","last_undegraded":"2024-09-22T03:23:05.009138+0000","last_fullsized":"2024-09-22T03:23:05.009138+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:21:11.969997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:21:11.969997+0000","last_clean_scrub_stamp":"2024-09-22T03:21:11.969997+0000","objects_scrubbed":0,"log_size":344,"log_dups_size":0,"ondisk_log_size":344,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2818048,"num_objects":43,"num_object_clones":0,"num_object_copies":129,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":344,"num_write_kb":2752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":291,"reported_epoch":169,"state":"peering","last_fresh":"2024-09-22T03:22:42.038566+0000","last_change":"2024-09-22T03:23:03.996221+0000","last_active":"2024-09-22T03:23:03.996221+0000","last_peered":"2024-09-22T03:22:42.038566+0000","last_clean":"2024-09-22T03:22:42.038566+0000","last_became_active":"2024-09-22T03:22:30.068029+0000","last_became_peered":"2024-09-22T03:22:30.068029+0000","last_unstale":"2024-09-22T03:22:42.038566+0000","last_undegraded":"2024-09-22T03:22:42.038566+0000","last_fullsized":"2024-09-22T03:22:42.038566+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"12'97","reported_seq":496,"reported_epoch":169,"state":"peering","last_fresh":"2024-09-22T03:22:42.073238+0000","last_change":"2024-09-22T03:23:03.990603+0000","last_active":"2024-09-22T03:23:03.990603+0000","last_peered":"2024-09-22T03:22:42.073238+0000","last_clean":"2024-09-22T03:22:42.073238+0000","last_became_active":"2024-09-22T03:22:30.088734+0000","last_became_peered":"2024-09-22T03:22:30.088734+0000","last_unstale":"2024-09-22T03:22:42.073238+0000","last_undegraded":"2024-09-22T03:22:42.073238+0000","last_fullsized":"2024-09-22T03:22:42.073238+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.f","version":"168'384","reported_seq":468,"reported_epoch":169,"state":"peering","last_fresh":"2024-09-22T03:22:42.574234+0000","last_change":"2024-09-22T03:23:04.004211+0000","last_active":"2024-09-22T03:23:04.004211+0000","last_peered":"2024-09-22T03:22:42.574234+0000","last_clean":"2024-09-22T03:22:42.574234+0000","last_became_active":"2024-09-22T03:22:30.054165+0000","last_became_peered":"2024-09-22T03:22:30.054165+0000","last_unstale":"2024-09-22T03:22:42.574234+0000","last_undegraded":"2024-09-22T03:22:42.574234+0000","last_fullsized":"2024-09-22T03:22:42.574234+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:21:11.969997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:21:11.969997+0000","last_clean_scrub_stamp":"2024-09-22T03:21:11.969997+0000","objects_scrubbed":0,"log_size":384,"log_dups_size":0,"ondisk_log_size":384,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3145728,"num_objects":48,"num_object_clones":0,"num_object_copies":144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":384,"num_write_kb":3072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":786432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":316,"reported_epoch":169,"state":"peering","last_fresh":"2024-09-22T03:22:30.012793+0000","last_change":"2024-09-22T03:22:29.008205+0000","last_active":"2024-09-22T03:22:29.008205+0000","last_peered":"2024-09-22T03:22:26.042599+0000","last_clean":"2024-09-22T03:21:50.022603+0000","last_became_active":"2024-09-22T03:22:26.035575+0000","last_became_peered":"2024-09-22T03:22:26.035575+0000","last_unstale":"2024-09-22T03:22:30.012793+0000","last_undegraded":"2024-09-22T03:22:30.012793+0000","last_fullsized":"2024-09-22T03:22:30.012793+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.e","version":"168'424","reported_seq":414,"reported_epoch":169,"state":"peering","last_fresh":"2024-09-22T03:22:42.978382+0000","last_change":"2024-09-22T03:23:04.005740+0000","last_active":"2024-09-22T03:23:04.005740+0000","last_peered":"2024-09-22T03:22:42.978382+0000","last_clean":"2024-09-22T03:22:42.978382+0000","last_became_active":"2024-09-22T03:22:30.135157+0000","last_became_peered":"2024-09-22T03:22:30.135157+0000","last_unstale":"2024-09-22T03:22:42.978382+0000","last_undegraded":"2024-09-22T03:22:42.978382+0000","last_fullsized":"2024-09-22T03:22:42.978382+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:21:11.969997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:21:11.969997+0000","last_clean_scrub_stamp":"2024-09-22T03:21:11.969997+0000","objects_scrubbed":0,"log_size":424,"log_dups_size":0,"ondisk_log_size":424,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3473408,"num_objects":53,"num_object_clones":0,"num_object_copies":159,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":424,"num_write_kb":3392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":379,"reported_epoch":170,"state":"active+clean","last_fresh":"2024-09-22T03:23:05.007477+0000","last_change":"2024-09-22T03:22:39.026321+0000","last_active":"2024-09-22T03:23:05.007477+0000","last_peered":"2024-09-22T03:23:05.007477+0000","last_clean":"2024-09-22T03:23:05.007477+0000","last_became_active":"2024-09-22T03:22:30.055276+0000","last_became_peered":"2024-09-22T03:22:30.055276+0000","last_unstale":"2024-09-22T03:23:05.007477+0000","last_undegraded":"2024-09-22T03:23:05.007477+0000","last_fullsized":"2024-09-22T03:23:05.007477+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.d","version":"168'393","reported_seq":383,"reported_epoch":169,"state":"peering","last_fresh":"2024-09-22T03:22:43.036184+0000","last_change":"2024-09-22T03:23:04.014941+0000","last_active":"2024-09-22T03:23:04.014941+0000","last_peered":"2024-09-22T03:22:43.036184+0000","last_clean":"2024-09-22T03:22:43.036184+0000","last_became_active":"2024-09-22T03:22:30.111068+0000","last_became_peered":"2024-09-22T03:22:30.111068+0000","last_unstale":"2024-09-22T03:22:43.036184+0000","last_undegraded":"2024-09-22T03:22:43.036184+0000","last_fullsized":"2024-09-22T03:22:43.036184+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:21:11.969997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:21:11.969997+0000","last_clean_scrub_stamp":"2024-09-22T03:21:11.969997+0000","objects_scrubbed":0,"log_size":393,"log_dups_size":0,"ondisk_log_size":393,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3219456,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":393,"num_write_kb":3144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":381,"reported_epoch":169,"state":"peering","last_fresh":"2024-09-22T03:22:16.362442+0000","last_change":"2024-09-22T03:22:15.374847+0000","last_active":"2024-09-22T03:22:15.374847+0000","last_peered":"2024-09-22T03:21:58.016704+0000","last_clean":"2024-09-22T03:21:13.969925+0000","last_became_active":"2024-09-22T03:21:58.013769+0000","last_became_peered":"2024-09-22T03:21:58.013769+0000","last_unstale":"2024-09-22T03:22:16.362442+0000","last_undegraded":"2024-09-22T03:22:16.362442+0000","last_fullsized":"2024-09-22T03:22:16.362442+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.c","version":"168'408","reported_seq":400,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-22T03:23:04.009739+0000","last_change":"2024-09-22T03:22:30.382901+0000","last_active":"2024-09-22T03:23:04.009739+0000","last_peered":"2024-09-22T03:23:04.009739+0000","last_clean":"2024-09-22T03:23:04.009739+0000","last_became_active":"2024-09-22T03:22:30.094983+0000","last_became_peered":"2024-09-22T03:22:30.094983+0000","last_unstale":"2024-09-22T03:23:04.009739+0000","last_undegraded":"2024-09-22T03:23:04.009739+0000","last_fullsized":"2024-09-22T03:23:04.009739+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:21:11.969997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:21:11.969997+0000","last_clean_scrub_stamp":"2024-09-22T03:21:11.969997+0000","objects_scrubbed":0,"log_size":408,"log_dups_size":0,"ondisk_log_size":408,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3342336,"num_objects":51,"num_object_clones":0,"num_object_copies":153,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":408,"num_write_kb":3264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":16384,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":401,"reported_epoch":169,"state":"peering","last_fresh":"2024-09-22T03:22:30.011668+0000","last_change":"2024-09-22T03:22:29.021046+0000","last_active":"2024-09-22T03:22:29.021046+0000","last_peered":"2024-09-22T03:22:26.018302+0000","last_clean":"2024-09-22T03:22:16.362770+0000","last_became_active":"2024-09-22T03:22:26.016006+0000","last_became_peered":"2024-09-22T03:22:26.016006+0000","last_unstale":"2024-09-22T03:22:30.011668+0000","last_undegraded":"2024-09-22T03:22:30.011668+0000","last_fullsized":"2024-09-22T03:22:30.011668+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":16,"stat_sum":{"num_bytes":55074816,"num_objects":842,"num_object_clones":0,"num_object_copies":2526,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6723,"num_write_kb":53784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":112,"num_bytes_recovered":3170304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":79114240,"data_stored":164544512,"data_compressed":8465041,"data_compressed_allocated":79114240,"data_compressed_original":164544512,"omap_allocated":0,"internal_metadata":0},"log_size":6722,"ondisk_log_size":6722,"up":38,"acting":38,"num_store_stats":4},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1180736,"data_compressed":59924,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":19,"acting":19,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":155,"seq":665719930955,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":51364,"kb_used_data":16468,"kb_used_omap":0,"kb_used_meta":34880,"kb_avail":94320476,"statfs":{"total":96636764160,"available":96584167424,"internally_reserved":0,"allocated":16863232,"data_stored":31509277,"data_compressed":1596346,"data_compressed_allocated":15441920,"data_compressed_original":30932992,"omap_allocated":0,"internal_metadata":35717120},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":159,"seq":682899800114,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":62684,"kb_used_data":25228,"kb_used_omap":0,"kb_used_meta":37440,"kb_avail":94309156,"statfs":{"total":96636764160,"available":96572575744,"internally_reserved":0,"allocated":25833472,"data_stored":49888403,"data_compressed":2544429,"data_compressed_allocated":24383488,"data_compressed_original":49291264,"omap_allocated":0,"internal_metadata":38338560},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":138,"seq":592705487042,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":72456,"kb_used_data":20856,"kb_used_omap":0,"kb_used_meta":51584,"kb_avail":94299384,"statfs":{"total":96636764160,"available":96562569216,"internally_reserved":0,"allocated":21356544,"data_stored":40926949,"data_compressed":2082334,"data_compressed_allocated":19922944,"data_compressed_original":40337408,"omap_allocated":0,"internal_metadata":52822016},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":163,"seq":700079669283,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":51708,"kb_used_data":19756,"kb_used_omap":0,"kb_used_meta":31936,"kb_avail":94320132,"statfs":{"total":96636764160,"available":96583815168,"internally_reserved":0,"allocated":20230144,"data_stored":44903722,"data_compressed":2259908,"data_compressed_allocated":18792448,"data_compressed_original":44310528,"omap_allocated":0,"internal_metadata":32702464},"hb_peers":[2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29962,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29962,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":18751488,"data_stored":43737088,"data_compressed":2229946,"data_compressed_allocated":18751488,"data_compressed_original":43737088,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20578304,"data_stored":41156608,"data_compressed":2124282,"data_compressed_allocated":20578304,"data_compressed_original":41156608,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":24342528,"data_stored":48717824,"data_compressed":2514467,"data_compressed_allocated":24342528,"data_compressed_original":48717824,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":15441920,"data_stored":30932992,"data_compressed":1596346,"data_compressed_allocated":15441920,"data_compressed_original":30932992,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-22T03:23:09.934 INFO:tasks.ceph.ceph_manager.ceph:removing pool_name unique_pool_7 2024-09-22T03:23:09.934 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm unique_pool_7 unique_pool_7 --yes-i-really-really-mean-it 2024-09-22T03:23:10.046 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:23:10.037+0000 7f005b597640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:23:10.049 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:23:10.047+0000 7f005b597640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:23:10.730 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-22T03:23:10.846 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:23:10.843+0000 7f682706c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:23:10.849 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:23:10.848+0000 7f682706c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:23:11.003 INFO:teuthology.orchestra.run.smithi153.stderr:pool 'unique_pool_7' removed 2024-09-22T03:23:11.021 DEBUG:teuthology.run_tasks:Unwinding manager thrashosds 2024-09-22T03:23:11.030 INFO:tasks.thrashosds:joining thrashosds 2024-09-22T03:23:14.726 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=600 2024-09-22T03:23:14.841 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:23:14.838+0000 7f89584b2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:23:14.844 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:23:14.843+0000 7f89584b2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:23:14.857 INFO:teuthology.orchestra.run.smithi153.stdout:{} 2024-09-22T03:23:14.857 INFO:teuthology.orchestra.run.smithi153.stderr:mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-22T03:23:14.868 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-09-22T03:23:14.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:23:14.966+0000 7f462b797640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:23:14.976 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:23:14.975+0000 7f462b797640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:23:16.009 INFO:teuthology.orchestra.run.smithi153.stderr:reweighted osd.3 to 1 (10000) 2024-09-22T03:23:16.026 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 3 1 2024-09-22T03:23:16.134 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:23:16.127+0000 7f4bc6757640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:23:16.137 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:23:16.136+0000 7f4bc6757640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:23:17.014 INFO:teuthology.orchestra.run.smithi153.stderr:set osd.3 primary-affinity to 1 (8655362) 2024-09-22T03:23:17.032 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-09-22T03:23:17.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:23:17.135+0000 7f0d70d17640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:23:17.155 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:23:17.143+0000 7f0d70d17640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:23:18.021 INFO:teuthology.orchestra.run.smithi153.stderr:reweighted osd.2 to 1 (10000) 2024-09-22T03:23:18.039 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 2 1 2024-09-22T03:23:18.148 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:23:18.141+0000 7efeb6d57640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:23:18.151 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:23:18.150+0000 7efeb6d57640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:23:18.977 INFO:teuthology.orchestra.run.smithi153.stderr:set osd.2 primary-affinity to 1 (8655362) 2024-09-22T03:23:18.995 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-09-22T03:23:19.105 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:23:19.097+0000 7fdcd9597640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:23:19.107 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:23:19.106+0000 7fdcd9597640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:23:20.026 INFO:teuthology.orchestra.run.smithi153.stderr:reweighted osd.0 to 1 (10000) 2024-09-22T03:23:20.045 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 0 1 2024-09-22T03:23:20.155 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:23:20.148+0000 7fa3b2b97640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:23:20.158 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:23:20.156+0000 7fa3b2b97640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:23:21.030 INFO:teuthology.orchestra.run.smithi153.stderr:set osd.0 primary-affinity to 1 (8655362) 2024-09-22T03:23:21.047 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-09-22T03:23:21.153 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:23:21.146+0000 7f0b66d57640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:23:21.156 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:23:21.155+0000 7f0b66d57640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:23:22.034 INFO:teuthology.orchestra.run.smithi153.stderr:reweighted osd.1 to 1 (10000) 2024-09-22T03:23:22.055 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 1 1 2024-09-22T03:23:22.166 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:23:22.159+0000 7f69fa997640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:23:22.170 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:23:22.168+0000 7f69fa997640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:23:23.039 INFO:teuthology.orchestra.run.smithi153.stderr:set osd.1 primary-affinity to 1 (8655362) 2024-09-22T03:23:23.058 INFO:tasks.thrashosds.thrasher:joining the do_optrack_toggle greenlet 2024-09-22T03:23:23.058 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-09-22T03:23:23.059 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-22T03:23:23.166 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:23:23.160+0000 7fc83bafa640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:23:23.169 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:23:23.167+0000 7fc83bafa640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:23:23.345 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-22T03:23:23.345 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":182,"fsid":"f17abef2-bc08-46d2-abf1-d47a95f0e2ee","created":"2024-09-22T03:07:53.859391+0000","modified":"2024-09-22T03:23:23.037949+0000","last_up_change":"2024-09-22T03:23:04.979224+0000","last_in_change":"2024-09-22T03:07:55.944112+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2024-09-22T03:08:04.049314+0000","flags":270369,"flags_names":"hashpspool,nopgchange,selfmanaged_snaps,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":14,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".mgr","create_time":"2024-09-22T03:08:04.133626+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.75,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"35bc7c89-cb7c-4664-bed6-4088394939d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":142,"last_clean_end":156,"up_from":163,"up_thru":170,"down_at":161,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6803","nonce":4121670064}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":4121670064}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6805","nonce":4121670064}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":4121670064}]},"public_addr":"172.21.15.153:6803/4121670064","cluster_addr":"172.21.15.153:6804/4121670064","heartbeat_back_addr":"172.21.15.153:6805/4121670064","heartbeat_front_addr":"172.21.15.153:6806/4121670064","state":["exists","up"]},{"osd":1,"uuid":"513ab51d-4bac-4f1a-ade1-755e26e4c8e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":138,"last_clean_end":168,"up_from":170,"up_thru":170,"down_at":169,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6807","nonce":1104835390}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":1104835390}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6801","nonce":1104835390}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":1104835390}]},"public_addr":"172.21.15.153:6807/1104835390","cluster_addr":"172.21.15.153:6802/1104835390","heartbeat_back_addr":"172.21.15.153:6801/1104835390","heartbeat_front_addr":"172.21.15.153:6800/1104835390","state":["exists","up"]},{"osd":2,"uuid":"dd57bc3a-b76e-4c7f-8935-6c2b7eb8ef7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":100,"last_clean_end":152,"up_from":159,"up_thru":170,"down_at":157,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6810","nonce":2571484571}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6811","nonce":2571484571}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6813","nonce":2571484571}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6814","nonce":2571484571}]},"public_addr":"172.21.15.196:6810/2571484571","cluster_addr":"172.21.15.196:6811/2571484571","heartbeat_back_addr":"172.21.15.196:6813/2571484571","heartbeat_front_addr":"172.21.15.196:6814/2571484571","state":["exists","up"]},{"osd":3,"uuid":"76e7924f-5812-47ee-a5ee-ec837d12f327","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":146,"last_clean_end":152,"up_from":155,"up_thru":170,"down_at":153,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6803","nonce":126290868}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6804","nonce":126290868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6806","nonce":126290868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6807","nonce":126290868}]},"public_addr":"172.21.15.196:6803/126290868","cluster_addr":"172.21.15.196:6804/126290868","heartbeat_back_addr":"172.21.15.196:6806/126290868","heartbeat_front_addr":"172.21.15.196:6807/126290868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"2024-09-22T03:22:24.973236+0000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"2024-09-22T03:23:03.976941+0000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"2024-09-22T03:21:56.968452+0000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"2024-09-22T03:21:44.981530+0000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":1,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-22T03:23:23.359 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-09-22T03:23:23.359 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-22T03:23:23.463 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:23:23.458+0000 7f3cd5a2c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:23:23.466 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:23:23.465+0000 7f3cd5a2c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:23:23.644 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-22T03:23:23.645 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":182,"fsid":"f17abef2-bc08-46d2-abf1-d47a95f0e2ee","created":"2024-09-22T03:07:53.859391+0000","modified":"2024-09-22T03:23:23.037949+0000","last_up_change":"2024-09-22T03:23:04.979224+0000","last_in_change":"2024-09-22T03:07:55.944112+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2024-09-22T03:08:04.049314+0000","flags":270369,"flags_names":"hashpspool,nopgchange,selfmanaged_snaps,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":14,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".mgr","create_time":"2024-09-22T03:08:04.133626+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.75,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"35bc7c89-cb7c-4664-bed6-4088394939d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":142,"last_clean_end":156,"up_from":163,"up_thru":170,"down_at":161,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6803","nonce":4121670064}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":4121670064}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6805","nonce":4121670064}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":4121670064}]},"public_addr":"172.21.15.153:6803/4121670064","cluster_addr":"172.21.15.153:6804/4121670064","heartbeat_back_addr":"172.21.15.153:6805/4121670064","heartbeat_front_addr":"172.21.15.153:6806/4121670064","state":["exists","up"]},{"osd":1,"uuid":"513ab51d-4bac-4f1a-ade1-755e26e4c8e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":138,"last_clean_end":168,"up_from":170,"up_thru":170,"down_at":169,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6807","nonce":1104835390}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":1104835390}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6801","nonce":1104835390}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":1104835390}]},"public_addr":"172.21.15.153:6807/1104835390","cluster_addr":"172.21.15.153:6802/1104835390","heartbeat_back_addr":"172.21.15.153:6801/1104835390","heartbeat_front_addr":"172.21.15.153:6800/1104835390","state":["exists","up"]},{"osd":2,"uuid":"dd57bc3a-b76e-4c7f-8935-6c2b7eb8ef7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":100,"last_clean_end":152,"up_from":159,"up_thru":170,"down_at":157,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6810","nonce":2571484571}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6811","nonce":2571484571}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6813","nonce":2571484571}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6814","nonce":2571484571}]},"public_addr":"172.21.15.196:6810/2571484571","cluster_addr":"172.21.15.196:6811/2571484571","heartbeat_back_addr":"172.21.15.196:6813/2571484571","heartbeat_front_addr":"172.21.15.196:6814/2571484571","state":["exists","up"]},{"osd":3,"uuid":"76e7924f-5812-47ee-a5ee-ec837d12f327","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":146,"last_clean_end":152,"up_from":155,"up_thru":170,"down_at":153,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6803","nonce":126290868}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6804","nonce":126290868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6806","nonce":126290868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6807","nonce":126290868}]},"public_addr":"172.21.15.196:6803/126290868","cluster_addr":"172.21.15.196:6804/126290868","heartbeat_back_addr":"172.21.15.196:6806/126290868","heartbeat_front_addr":"172.21.15.196:6807/126290868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"2024-09-22T03:22:24.973236+0000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"2024-09-22T03:23:03.976941+0000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"2024-09-22T03:21:56.968452+0000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"2024-09-22T03:21:44.981530+0000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":1,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-22T03:23:23.658 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-09-22T03:23:23.659 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-09-22T03:23:23.659 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-09-22T03:23:23.660 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-09-22T03:23:23.775 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:23:23.769+0000 7fe2642fa640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:23:23.775 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:23:23.769+0000 7fb10fa6c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:23:23.778 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:23:23.769+0000 7ffa2cc23640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:23:23.778 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:23:23.769+0000 7f47890ce640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:23:23.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:23:23.777+0000 7fb10fa6c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:23:23.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:23:23.777+0000 7fe2642fa640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:23:23.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:23:23.778+0000 7ffa2cc23640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:23:23.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:23:23.778+0000 7f47890ce640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:23:23.811 INFO:teuthology.orchestra.run.smithi153.stdout:665719930974 2024-09-22T03:23:23.812 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-22T03:23:23.812 INFO:teuthology.orchestra.run.smithi153.stdout:682899800132 2024-09-22T03:23:23.813 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-22T03:23:23.816 INFO:teuthology.orchestra.run.smithi153.stdout:730144440338 2024-09-22T03:23:23.816 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-22T03:23:23.817 INFO:teuthology.orchestra.run.smithi153.stdout:700079669302 2024-09-22T03:23:23.817 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-22T03:23:23.927 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:23:23.919+0000 7fe399397640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:23:23.933 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:23:23.931+0000 7fe399397640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:23:23.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:23:23.963+0000 7f5eea82c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:23:23.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:23:23.964+0000 7faffc397640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:23:23.974 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:23:23.972+0000 7faffc397640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:23:23.974 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:23:23.972+0000 7f5eea82c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:23:23.975 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:23:23.968+0000 7f08a4f97640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:23:23.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:23:23.987+0000 7f08a4f97640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:23:24.130 INFO:teuthology.orchestra.run.smithi153.stdout:665719930970 2024-09-22T03:23:24.147 INFO:tasks.ceph.ceph_manager.ceph:need seq 665719930974 got 665719930970 for osd.3 2024-09-22T03:23:24.171 INFO:teuthology.orchestra.run.smithi153.stdout:682899800129 2024-09-22T03:23:24.182 INFO:teuthology.orchestra.run.smithi153.stdout:730144440334 2024-09-22T03:23:24.184 INFO:teuthology.orchestra.run.smithi153.stdout:700079669298 2024-09-22T03:23:24.186 INFO:tasks.ceph.ceph_manager.ceph:need seq 682899800132 got 682899800129 for osd.2 2024-09-22T03:23:24.197 INFO:tasks.ceph.ceph_manager.ceph:need seq 730144440338 got 730144440334 for osd.1 2024-09-22T03:23:24.198 INFO:tasks.ceph.ceph_manager.ceph:need seq 700079669302 got 700079669298 for osd.0 2024-09-22T03:23:25.148 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-22T03:23:25.187 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-22T03:23:25.197 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-22T03:23:25.199 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-22T03:23:25.270 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:23:25.262+0000 7faa8b82c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:23:25.275 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:23:25.273+0000 7faa8b82c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:23:25.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:23:25.309+0000 7fee94ed7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:23:25.322 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:23:25.320+0000 7fee94ed7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:23:25.350 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:23:25.343+0000 7f21928d7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:23:25.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:23:25.351+0000 7f21928d7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:23:25.358 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:23:25.351+0000 7f7b67397640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:23:25.364 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:23:25.362+0000 7f7b67397640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:23:25.512 INFO:teuthology.orchestra.run.smithi153.stdout:665719930975 2024-09-22T03:23:25.528 INFO:tasks.ceph.ceph_manager.ceph:need seq 665719930974 got 665719930975 for osd.3 2024-09-22T03:23:25.528 DEBUG:teuthology.parallel:result is None 2024-09-22T03:23:25.543 INFO:teuthology.orchestra.run.smithi153.stdout:700079669302 2024-09-22T03:23:25.550 INFO:teuthology.orchestra.run.smithi153.stdout:682899800132 2024-09-22T03:23:25.552 INFO:teuthology.orchestra.run.smithi153.stdout:730144440338 2024-09-22T03:23:25.558 INFO:tasks.ceph.ceph_manager.ceph:need seq 700079669302 got 700079669302 for osd.0 2024-09-22T03:23:25.558 DEBUG:teuthology.parallel:result is None 2024-09-22T03:23:25.565 INFO:tasks.ceph.ceph_manager.ceph:need seq 682899800132 got 682899800132 for osd.2 2024-09-22T03:23:25.565 DEBUG:teuthology.parallel:result is None 2024-09-22T03:23:25.566 INFO:tasks.ceph.ceph_manager.ceph:need seq 730144440338 got 730144440338 for osd.1 2024-09-22T03:23:25.566 DEBUG:teuthology.parallel:result is None 2024-09-22T03:23:25.567 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-09-22T03:23:25.567 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-22T03:23:25.711 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:23:25.706+0000 7f4061b97640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:23:25.714 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:23:25.713+0000 7f4061b97640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:23:25.894 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-22T03:23:25.895 INFO:teuthology.orchestra.run.smithi153.stderr:dumped all 2024-09-22T03:23:25.907 INFO:teuthology.orchestra.run.smithi153.stdout:{"pg_ready":true,"pg_map":{"version":636,"stamp":"2024-09-22T03:23:24.303060+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":139,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":27,"acting":27,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":161344,"kb_used_data":6144,"kb_used_omap":0,"kb_used_meta":155136,"kb_avail":377326016,"statfs":{"total":386547056640,"available":386381840384,"internally_reserved":0,"allocated":6291456,"data_stored":4230477,"data_compressed":89886,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":158859264},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.276882"},"pg_stats":[{"pgid":"1.0","version":"0'0","reported_seq":284,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-22T03:23:23.046203+0000","last_change":"2024-09-22T03:22:39.015859+0000","last_active":"2024-09-22T03:23:23.046203+0000","last_peered":"2024-09-22T03:23:23.046203+0000","last_clean":"2024-09-22T03:23:23.046203+0000","last_became_active":"2024-09-22T03:22:30.096861+0000","last_became_peered":"2024-09-22T03:22:30.096861+0000","last_unstale":"2024-09-22T03:23:23.046203+0000","last_undegraded":"2024-09-22T03:23:23.046203+0000","last_fullsized":"2024-09-22T03:23:23.046203+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":419,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-22T03:23:23.048353+0000","last_change":"2024-09-22T03:23:08.987134+0000","last_active":"2024-09-22T03:23:23.048353+0000","last_peered":"2024-09-22T03:23:23.048353+0000","last_clean":"2024-09-22T03:23:23.048353+0000","last_became_active":"2024-09-22T03:23:06.048233+0000","last_became_peered":"2024-09-22T03:23:06.048233+0000","last_unstale":"2024-09-22T03:23:23.048353+0000","last_undegraded":"2024-09-22T03:23:23.048353+0000","last_fullsized":"2024-09-22T03:23:23.048353+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":447,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-22T03:23:23.046737+0000","last_change":"2024-09-22T03:23:08.992662+0000","last_active":"2024-09-22T03:23:23.046737+0000","last_peered":"2024-09-22T03:23:23.046737+0000","last_clean":"2024-09-22T03:23:23.046737+0000","last_became_active":"2024-09-22T03:23:06.069718+0000","last_became_peered":"2024-09-22T03:23:06.069718+0000","last_unstale":"2024-09-22T03:23:23.046737+0000","last_undegraded":"2024-09-22T03:23:23.046737+0000","last_fullsized":"2024-09-22T03:23:23.046737+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":334,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-22T03:23:23.047694+0000","last_change":"2024-09-22T03:23:08.993326+0000","last_active":"2024-09-22T03:23:23.047694+0000","last_peered":"2024-09-22T03:23:23.047694+0000","last_clean":"2024-09-22T03:23:23.047694+0000","last_became_active":"2024-09-22T03:23:06.050619+0000","last_became_peered":"2024-09-22T03:23:06.050619+0000","last_unstale":"2024-09-22T03:23:23.047694+0000","last_undegraded":"2024-09-22T03:23:23.047694+0000","last_fullsized":"2024-09-22T03:23:23.047694+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"12'97","reported_seq":538,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-22T03:23:23.047931+0000","last_change":"2024-09-22T03:23:06.061505+0000","last_active":"2024-09-22T03:23:23.047931+0000","last_peered":"2024-09-22T03:23:23.047931+0000","last_clean":"2024-09-22T03:23:23.047931+0000","last_became_active":"2024-09-22T03:23:06.050659+0000","last_became_peered":"2024-09-22T03:23:06.050659+0000","last_unstale":"2024-09-22T03:23:23.047931+0000","last_undegraded":"2024-09-22T03:23:23.047931+0000","last_fullsized":"2024-09-22T03:23:23.047931+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":365,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-22T03:23:23.047132+0000","last_change":"2024-09-22T03:23:09.008207+0000","last_active":"2024-09-22T03:23:23.047132+0000","last_peered":"2024-09-22T03:23:23.047132+0000","last_clean":"2024-09-22T03:23:23.047132+0000","last_became_active":"2024-09-22T03:23:06.123987+0000","last_became_peered":"2024-09-22T03:23:06.123987+0000","last_unstale":"2024-09-22T03:23:23.047132+0000","last_undegraded":"2024-09-22T03:23:23.047132+0000","last_fullsized":"2024-09-22T03:23:23.047132+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":415,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-22T03:23:23.046206+0000","last_change":"2024-09-22T03:22:39.026321+0000","last_active":"2024-09-22T03:23:23.046206+0000","last_peered":"2024-09-22T03:23:23.046206+0000","last_clean":"2024-09-22T03:23:23.046206+0000","last_became_active":"2024-09-22T03:22:30.055276+0000","last_became_peered":"2024-09-22T03:22:30.055276+0000","last_unstale":"2024-09-22T03:23:23.046206+0000","last_undegraded":"2024-09-22T03:23:23.046206+0000","last_fullsized":"2024-09-22T03:23:23.046206+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":430,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-22T03:23:23.046759+0000","last_change":"2024-09-22T03:23:09.006611+0000","last_active":"2024-09-22T03:23:23.046759+0000","last_peered":"2024-09-22T03:23:23.046759+0000","last_clean":"2024-09-22T03:23:23.046759+0000","last_became_active":"2024-09-22T03:23:06.062711+0000","last_became_peered":"2024-09-22T03:23:06.062711+0000","last_unstale":"2024-09-22T03:23:23.046759+0000","last_undegraded":"2024-09-22T03:23:23.046759+0000","last_fullsized":"2024-09-22T03:23:23.046759+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":450,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-22T03:23:23.047579+0000","last_change":"2024-09-22T03:23:09.006281+0000","last_active":"2024-09-22T03:23:23.047579+0000","last_peered":"2024-09-22T03:23:23.047579+0000","last_clean":"2024-09-22T03:23:23.047579+0000","last_became_active":"2024-09-22T03:23:06.094384+0000","last_became_peered":"2024-09-22T03:23:06.094384+0000","last_unstale":"2024-09-22T03:23:23.047579+0000","last_undegraded":"2024-09-22T03:23:23.047579+0000","last_fullsized":"2024-09-22T03:23:23.047579+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1180736,"data_compressed":59924,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":155,"seq":665719930975,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":40552,"kb_used_data":1496,"kb_used_omap":0,"kb_used_meta":39040,"kb_avail":94331288,"statfs":{"total":96636764160,"available":96595238912,"internally_reserved":0,"allocated":1531904,"data_stored":616310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":39976960},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":159,"seq":682899800132,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":43172,"kb_used_data":1556,"kb_used_omap":0,"kb_used_meta":41600,"kb_avail":94328668,"statfs":{"total":96636764160,"available":96592556032,"internally_reserved":0,"allocated":1593344,"data_stored":1206678,"data_compressed":29962,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":42598400},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":170,"seq":730144440338,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":39964,"kb_used_data":1548,"kb_used_omap":0,"kb_used_meta":38400,"kb_avail":94331876,"statfs":{"total":96636764160,"available":96595841024,"internally_reserved":0,"allocated":1585152,"data_stored":1203754,"data_compressed":29962,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":39321600},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":163,"seq":700079669302,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":37656,"kb_used_data":1544,"kb_used_omap":0,"kb_used_meta":36096,"kb_avail":94334184,"statfs":{"total":96636764160,"available":96598204416,"internally_reserved":0,"allocated":1581056,"data_stored":1203735,"data_compressed":29962,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":36962304},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29962,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29962,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-22T03:23:25.908 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-22T03:23:26.011 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:23:26.007+0000 7faf90197640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:23:26.015 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:23:26.013+0000 7faf90197640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:23:26.195 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-22T03:23:26.195 INFO:teuthology.orchestra.run.smithi153.stderr:dumped all 2024-09-22T03:23:26.207 INFO:teuthology.orchestra.run.smithi153.stdout:{"pg_ready":true,"pg_map":{"version":636,"stamp":"2024-09-22T03:23:24.303060+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":139,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":27,"acting":27,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":161344,"kb_used_data":6144,"kb_used_omap":0,"kb_used_meta":155136,"kb_avail":377326016,"statfs":{"total":386547056640,"available":386381840384,"internally_reserved":0,"allocated":6291456,"data_stored":4230477,"data_compressed":89886,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":158859264},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.276882"},"pg_stats":[{"pgid":"1.0","version":"0'0","reported_seq":284,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-22T03:23:23.046203+0000","last_change":"2024-09-22T03:22:39.015859+0000","last_active":"2024-09-22T03:23:23.046203+0000","last_peered":"2024-09-22T03:23:23.046203+0000","last_clean":"2024-09-22T03:23:23.046203+0000","last_became_active":"2024-09-22T03:22:30.096861+0000","last_became_peered":"2024-09-22T03:22:30.096861+0000","last_unstale":"2024-09-22T03:23:23.046203+0000","last_undegraded":"2024-09-22T03:23:23.046203+0000","last_fullsized":"2024-09-22T03:23:23.046203+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":419,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-22T03:23:23.048353+0000","last_change":"2024-09-22T03:23:08.987134+0000","last_active":"2024-09-22T03:23:23.048353+0000","last_peered":"2024-09-22T03:23:23.048353+0000","last_clean":"2024-09-22T03:23:23.048353+0000","last_became_active":"2024-09-22T03:23:06.048233+0000","last_became_peered":"2024-09-22T03:23:06.048233+0000","last_unstale":"2024-09-22T03:23:23.048353+0000","last_undegraded":"2024-09-22T03:23:23.048353+0000","last_fullsized":"2024-09-22T03:23:23.048353+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":447,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-22T03:23:23.046737+0000","last_change":"2024-09-22T03:23:08.992662+0000","last_active":"2024-09-22T03:23:23.046737+0000","last_peered":"2024-09-22T03:23:23.046737+0000","last_clean":"2024-09-22T03:23:23.046737+0000","last_became_active":"2024-09-22T03:23:06.069718+0000","last_became_peered":"2024-09-22T03:23:06.069718+0000","last_unstale":"2024-09-22T03:23:23.046737+0000","last_undegraded":"2024-09-22T03:23:23.046737+0000","last_fullsized":"2024-09-22T03:23:23.046737+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":334,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-22T03:23:23.047694+0000","last_change":"2024-09-22T03:23:08.993326+0000","last_active":"2024-09-22T03:23:23.047694+0000","last_peered":"2024-09-22T03:23:23.047694+0000","last_clean":"2024-09-22T03:23:23.047694+0000","last_became_active":"2024-09-22T03:23:06.050619+0000","last_became_peered":"2024-09-22T03:23:06.050619+0000","last_unstale":"2024-09-22T03:23:23.047694+0000","last_undegraded":"2024-09-22T03:23:23.047694+0000","last_fullsized":"2024-09-22T03:23:23.047694+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"12'97","reported_seq":538,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-22T03:23:23.047931+0000","last_change":"2024-09-22T03:23:06.061505+0000","last_active":"2024-09-22T03:23:23.047931+0000","last_peered":"2024-09-22T03:23:23.047931+0000","last_clean":"2024-09-22T03:23:23.047931+0000","last_became_active":"2024-09-22T03:23:06.050659+0000","last_became_peered":"2024-09-22T03:23:06.050659+0000","last_unstale":"2024-09-22T03:23:23.047931+0000","last_undegraded":"2024-09-22T03:23:23.047931+0000","last_fullsized":"2024-09-22T03:23:23.047931+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":365,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-22T03:23:23.047132+0000","last_change":"2024-09-22T03:23:09.008207+0000","last_active":"2024-09-22T03:23:23.047132+0000","last_peered":"2024-09-22T03:23:23.047132+0000","last_clean":"2024-09-22T03:23:23.047132+0000","last_became_active":"2024-09-22T03:23:06.123987+0000","last_became_peered":"2024-09-22T03:23:06.123987+0000","last_unstale":"2024-09-22T03:23:23.047132+0000","last_undegraded":"2024-09-22T03:23:23.047132+0000","last_fullsized":"2024-09-22T03:23:23.047132+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":415,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-22T03:23:23.046206+0000","last_change":"2024-09-22T03:22:39.026321+0000","last_active":"2024-09-22T03:23:23.046206+0000","last_peered":"2024-09-22T03:23:23.046206+0000","last_clean":"2024-09-22T03:23:23.046206+0000","last_became_active":"2024-09-22T03:22:30.055276+0000","last_became_peered":"2024-09-22T03:22:30.055276+0000","last_unstale":"2024-09-22T03:23:23.046206+0000","last_undegraded":"2024-09-22T03:23:23.046206+0000","last_fullsized":"2024-09-22T03:23:23.046206+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":430,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-22T03:23:23.046759+0000","last_change":"2024-09-22T03:23:09.006611+0000","last_active":"2024-09-22T03:23:23.046759+0000","last_peered":"2024-09-22T03:23:23.046759+0000","last_clean":"2024-09-22T03:23:23.046759+0000","last_became_active":"2024-09-22T03:23:06.062711+0000","last_became_peered":"2024-09-22T03:23:06.062711+0000","last_unstale":"2024-09-22T03:23:23.046759+0000","last_undegraded":"2024-09-22T03:23:23.046759+0000","last_fullsized":"2024-09-22T03:23:23.046759+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":450,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-22T03:23:23.047579+0000","last_change":"2024-09-22T03:23:09.006281+0000","last_active":"2024-09-22T03:23:23.047579+0000","last_peered":"2024-09-22T03:23:23.047579+0000","last_clean":"2024-09-22T03:23:23.047579+0000","last_became_active":"2024-09-22T03:23:06.094384+0000","last_became_peered":"2024-09-22T03:23:06.094384+0000","last_unstale":"2024-09-22T03:23:23.047579+0000","last_undegraded":"2024-09-22T03:23:23.047579+0000","last_fullsized":"2024-09-22T03:23:23.047579+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1180736,"data_compressed":59924,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":155,"seq":665719930975,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":40552,"kb_used_data":1496,"kb_used_omap":0,"kb_used_meta":39040,"kb_avail":94331288,"statfs":{"total":96636764160,"available":96595238912,"internally_reserved":0,"allocated":1531904,"data_stored":616310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":39976960},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":159,"seq":682899800132,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":43172,"kb_used_data":1556,"kb_used_omap":0,"kb_used_meta":41600,"kb_avail":94328668,"statfs":{"total":96636764160,"available":96592556032,"internally_reserved":0,"allocated":1593344,"data_stored":1206678,"data_compressed":29962,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":42598400},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":170,"seq":730144440338,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":39964,"kb_used_data":1548,"kb_used_omap":0,"kb_used_meta":38400,"kb_avail":94331876,"statfs":{"total":96636764160,"available":96595841024,"internally_reserved":0,"allocated":1585152,"data_stored":1203754,"data_compressed":29962,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":39321600},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":163,"seq":700079669302,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":37656,"kb_used_data":1544,"kb_used_omap":0,"kb_used_meta":36096,"kb_avail":94334184,"statfs":{"total":96636764160,"available":96598204416,"internally_reserved":0,"allocated":1581056,"data_stored":1203735,"data_compressed":29962,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":36962304},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29962,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29962,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-22T03:23:26.208 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-09-22T03:23:26.208 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr_debug_aggressive_pg_num_changes 2024-09-22T03:23:26.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:23:26.306+0000 7fc54204f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:23:26.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:23:26.315+0000 7fc54204f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:23:26.512 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-09-22T03:23:26.526 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-22T03:23:26.527 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-22T03:23:26.631 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:23:26.626+0000 7f4bdf84f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:23:26.634 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:23:26.633+0000 7f4bdf84f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:23:26.807 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-22T03:23:26.807 INFO:teuthology.orchestra.run.smithi153.stderr:dumped all 2024-09-22T03:23:26.820 INFO:teuthology.orchestra.run.smithi153.stdout:{"pg_ready":true,"pg_map":{"version":637,"stamp":"2024-09-22T03:23:26.303433+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":139,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":27,"acting":27,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":161360,"kb_used_data":6160,"kb_used_omap":0,"kb_used_meta":155136,"kb_avail":377326000,"statfs":{"total":386547056640,"available":386381824000,"internally_reserved":0,"allocated":6307840,"data_stored":4236325,"data_compressed":89886,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":158859264},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001200"},"pg_stats":[{"pgid":"1.0","version":"0'0","reported_seq":284,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-22T03:23:23.046203+0000","last_change":"2024-09-22T03:22:39.015859+0000","last_active":"2024-09-22T03:23:23.046203+0000","last_peered":"2024-09-22T03:23:23.046203+0000","last_clean":"2024-09-22T03:23:23.046203+0000","last_became_active":"2024-09-22T03:22:30.096861+0000","last_became_peered":"2024-09-22T03:22:30.096861+0000","last_unstale":"2024-09-22T03:23:23.046203+0000","last_undegraded":"2024-09-22T03:23:23.046203+0000","last_fullsized":"2024-09-22T03:23:23.046203+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":419,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-22T03:23:23.048353+0000","last_change":"2024-09-22T03:23:08.987134+0000","last_active":"2024-09-22T03:23:23.048353+0000","last_peered":"2024-09-22T03:23:23.048353+0000","last_clean":"2024-09-22T03:23:23.048353+0000","last_became_active":"2024-09-22T03:23:06.048233+0000","last_became_peered":"2024-09-22T03:23:06.048233+0000","last_unstale":"2024-09-22T03:23:23.048353+0000","last_undegraded":"2024-09-22T03:23:23.048353+0000","last_fullsized":"2024-09-22T03:23:23.048353+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":447,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-22T03:23:23.046737+0000","last_change":"2024-09-22T03:23:08.992662+0000","last_active":"2024-09-22T03:23:23.046737+0000","last_peered":"2024-09-22T03:23:23.046737+0000","last_clean":"2024-09-22T03:23:23.046737+0000","last_became_active":"2024-09-22T03:23:06.069718+0000","last_became_peered":"2024-09-22T03:23:06.069718+0000","last_unstale":"2024-09-22T03:23:23.046737+0000","last_undegraded":"2024-09-22T03:23:23.046737+0000","last_fullsized":"2024-09-22T03:23:23.046737+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":334,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-22T03:23:23.047694+0000","last_change":"2024-09-22T03:23:08.993326+0000","last_active":"2024-09-22T03:23:23.047694+0000","last_peered":"2024-09-22T03:23:23.047694+0000","last_clean":"2024-09-22T03:23:23.047694+0000","last_became_active":"2024-09-22T03:23:06.050619+0000","last_became_peered":"2024-09-22T03:23:06.050619+0000","last_unstale":"2024-09-22T03:23:23.047694+0000","last_undegraded":"2024-09-22T03:23:23.047694+0000","last_fullsized":"2024-09-22T03:23:23.047694+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"12'97","reported_seq":538,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-22T03:23:23.047931+0000","last_change":"2024-09-22T03:23:06.061505+0000","last_active":"2024-09-22T03:23:23.047931+0000","last_peered":"2024-09-22T03:23:23.047931+0000","last_clean":"2024-09-22T03:23:23.047931+0000","last_became_active":"2024-09-22T03:23:06.050659+0000","last_became_peered":"2024-09-22T03:23:06.050659+0000","last_unstale":"2024-09-22T03:23:23.047931+0000","last_undegraded":"2024-09-22T03:23:23.047931+0000","last_fullsized":"2024-09-22T03:23:23.047931+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":365,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-22T03:23:23.047132+0000","last_change":"2024-09-22T03:23:09.008207+0000","last_active":"2024-09-22T03:23:23.047132+0000","last_peered":"2024-09-22T03:23:23.047132+0000","last_clean":"2024-09-22T03:23:23.047132+0000","last_became_active":"2024-09-22T03:23:06.123987+0000","last_became_peered":"2024-09-22T03:23:06.123987+0000","last_unstale":"2024-09-22T03:23:23.047132+0000","last_undegraded":"2024-09-22T03:23:23.047132+0000","last_fullsized":"2024-09-22T03:23:23.047132+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":415,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-22T03:23:23.046206+0000","last_change":"2024-09-22T03:22:39.026321+0000","last_active":"2024-09-22T03:23:23.046206+0000","last_peered":"2024-09-22T03:23:23.046206+0000","last_clean":"2024-09-22T03:23:23.046206+0000","last_became_active":"2024-09-22T03:22:30.055276+0000","last_became_peered":"2024-09-22T03:22:30.055276+0000","last_unstale":"2024-09-22T03:23:23.046206+0000","last_undegraded":"2024-09-22T03:23:23.046206+0000","last_fullsized":"2024-09-22T03:23:23.046206+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":430,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-22T03:23:23.046759+0000","last_change":"2024-09-22T03:23:09.006611+0000","last_active":"2024-09-22T03:23:23.046759+0000","last_peered":"2024-09-22T03:23:23.046759+0000","last_clean":"2024-09-22T03:23:23.046759+0000","last_became_active":"2024-09-22T03:23:06.062711+0000","last_became_peered":"2024-09-22T03:23:06.062711+0000","last_unstale":"2024-09-22T03:23:23.046759+0000","last_undegraded":"2024-09-22T03:23:23.046759+0000","last_fullsized":"2024-09-22T03:23:23.046759+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":450,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-22T03:23:23.047579+0000","last_change":"2024-09-22T03:23:09.006281+0000","last_active":"2024-09-22T03:23:23.047579+0000","last_peered":"2024-09-22T03:23:23.047579+0000","last_clean":"2024-09-22T03:23:23.047579+0000","last_became_active":"2024-09-22T03:23:06.094384+0000","last_became_peered":"2024-09-22T03:23:06.094384+0000","last_unstale":"2024-09-22T03:23:23.047579+0000","last_undegraded":"2024-09-22T03:23:23.047579+0000","last_fullsized":"2024-09-22T03:23:23.047579+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1180736,"data_compressed":59924,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":155,"seq":665719930975,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":40552,"kb_used_data":1496,"kb_used_omap":0,"kb_used_meta":39040,"kb_avail":94331288,"statfs":{"total":96636764160,"available":96595238912,"internally_reserved":0,"allocated":1531904,"data_stored":616310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":39976960},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":159,"seq":682899800134,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":43172,"kb_used_data":1556,"kb_used_omap":0,"kb_used_meta":41600,"kb_avail":94328668,"statfs":{"total":96636764160,"available":96592556032,"internally_reserved":0,"allocated":1593344,"data_stored":1206678,"data_compressed":29962,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":42598400},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":170,"seq":730144440339,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":39972,"kb_used_data":1556,"kb_used_omap":0,"kb_used_meta":38400,"kb_avail":94331868,"statfs":{"total":96636764160,"available":96595832832,"internally_reserved":0,"allocated":1593344,"data_stored":1206678,"data_compressed":29962,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":39321600},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":163,"seq":700079669303,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":37664,"kb_used_data":1552,"kb_used_omap":0,"kb_used_meta":36096,"kb_avail":94334176,"statfs":{"total":96636764160,"available":96598196224,"internally_reserved":0,"allocated":1589248,"data_stored":1206659,"data_compressed":29962,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":36962304},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29962,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29962,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-22T03:23:26.821 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-22T03:23:26.924 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:23:26.919+0000 7fb203c4f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:23:26.927 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:23:26.926+0000 7fb203c4f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:23:27.103 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-22T03:23:27.104 INFO:teuthology.orchestra.run.smithi153.stderr:dumped all 2024-09-22T03:23:27.116 INFO:teuthology.orchestra.run.smithi153.stdout:{"pg_ready":true,"pg_map":{"version":637,"stamp":"2024-09-22T03:23:26.303433+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":139,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":27,"acting":27,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":161360,"kb_used_data":6160,"kb_used_omap":0,"kb_used_meta":155136,"kb_avail":377326000,"statfs":{"total":386547056640,"available":386381824000,"internally_reserved":0,"allocated":6307840,"data_stored":4236325,"data_compressed":89886,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":158859264},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001200"},"pg_stats":[{"pgid":"1.0","version":"0'0","reported_seq":284,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-22T03:23:23.046203+0000","last_change":"2024-09-22T03:22:39.015859+0000","last_active":"2024-09-22T03:23:23.046203+0000","last_peered":"2024-09-22T03:23:23.046203+0000","last_clean":"2024-09-22T03:23:23.046203+0000","last_became_active":"2024-09-22T03:22:30.096861+0000","last_became_peered":"2024-09-22T03:22:30.096861+0000","last_unstale":"2024-09-22T03:23:23.046203+0000","last_undegraded":"2024-09-22T03:23:23.046203+0000","last_fullsized":"2024-09-22T03:23:23.046203+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":419,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-22T03:23:23.048353+0000","last_change":"2024-09-22T03:23:08.987134+0000","last_active":"2024-09-22T03:23:23.048353+0000","last_peered":"2024-09-22T03:23:23.048353+0000","last_clean":"2024-09-22T03:23:23.048353+0000","last_became_active":"2024-09-22T03:23:06.048233+0000","last_became_peered":"2024-09-22T03:23:06.048233+0000","last_unstale":"2024-09-22T03:23:23.048353+0000","last_undegraded":"2024-09-22T03:23:23.048353+0000","last_fullsized":"2024-09-22T03:23:23.048353+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":447,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-22T03:23:23.046737+0000","last_change":"2024-09-22T03:23:08.992662+0000","last_active":"2024-09-22T03:23:23.046737+0000","last_peered":"2024-09-22T03:23:23.046737+0000","last_clean":"2024-09-22T03:23:23.046737+0000","last_became_active":"2024-09-22T03:23:06.069718+0000","last_became_peered":"2024-09-22T03:23:06.069718+0000","last_unstale":"2024-09-22T03:23:23.046737+0000","last_undegraded":"2024-09-22T03:23:23.046737+0000","last_fullsized":"2024-09-22T03:23:23.046737+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":334,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-22T03:23:23.047694+0000","last_change":"2024-09-22T03:23:08.993326+0000","last_active":"2024-09-22T03:23:23.047694+0000","last_peered":"2024-09-22T03:23:23.047694+0000","last_clean":"2024-09-22T03:23:23.047694+0000","last_became_active":"2024-09-22T03:23:06.050619+0000","last_became_peered":"2024-09-22T03:23:06.050619+0000","last_unstale":"2024-09-22T03:23:23.047694+0000","last_undegraded":"2024-09-22T03:23:23.047694+0000","last_fullsized":"2024-09-22T03:23:23.047694+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"12'97","reported_seq":538,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-22T03:23:23.047931+0000","last_change":"2024-09-22T03:23:06.061505+0000","last_active":"2024-09-22T03:23:23.047931+0000","last_peered":"2024-09-22T03:23:23.047931+0000","last_clean":"2024-09-22T03:23:23.047931+0000","last_became_active":"2024-09-22T03:23:06.050659+0000","last_became_peered":"2024-09-22T03:23:06.050659+0000","last_unstale":"2024-09-22T03:23:23.047931+0000","last_undegraded":"2024-09-22T03:23:23.047931+0000","last_fullsized":"2024-09-22T03:23:23.047931+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":365,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-22T03:23:23.047132+0000","last_change":"2024-09-22T03:23:09.008207+0000","last_active":"2024-09-22T03:23:23.047132+0000","last_peered":"2024-09-22T03:23:23.047132+0000","last_clean":"2024-09-22T03:23:23.047132+0000","last_became_active":"2024-09-22T03:23:06.123987+0000","last_became_peered":"2024-09-22T03:23:06.123987+0000","last_unstale":"2024-09-22T03:23:23.047132+0000","last_undegraded":"2024-09-22T03:23:23.047132+0000","last_fullsized":"2024-09-22T03:23:23.047132+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":415,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-22T03:23:23.046206+0000","last_change":"2024-09-22T03:22:39.026321+0000","last_active":"2024-09-22T03:23:23.046206+0000","last_peered":"2024-09-22T03:23:23.046206+0000","last_clean":"2024-09-22T03:23:23.046206+0000","last_became_active":"2024-09-22T03:22:30.055276+0000","last_became_peered":"2024-09-22T03:22:30.055276+0000","last_unstale":"2024-09-22T03:23:23.046206+0000","last_undegraded":"2024-09-22T03:23:23.046206+0000","last_fullsized":"2024-09-22T03:23:23.046206+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":430,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-22T03:23:23.046759+0000","last_change":"2024-09-22T03:23:09.006611+0000","last_active":"2024-09-22T03:23:23.046759+0000","last_peered":"2024-09-22T03:23:23.046759+0000","last_clean":"2024-09-22T03:23:23.046759+0000","last_became_active":"2024-09-22T03:23:06.062711+0000","last_became_peered":"2024-09-22T03:23:06.062711+0000","last_unstale":"2024-09-22T03:23:23.046759+0000","last_undegraded":"2024-09-22T03:23:23.046759+0000","last_fullsized":"2024-09-22T03:23:23.046759+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":450,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-22T03:23:23.047579+0000","last_change":"2024-09-22T03:23:09.006281+0000","last_active":"2024-09-22T03:23:23.047579+0000","last_peered":"2024-09-22T03:23:23.047579+0000","last_clean":"2024-09-22T03:23:23.047579+0000","last_became_active":"2024-09-22T03:23:06.094384+0000","last_became_peered":"2024-09-22T03:23:06.094384+0000","last_unstale":"2024-09-22T03:23:23.047579+0000","last_undegraded":"2024-09-22T03:23:23.047579+0000","last_fullsized":"2024-09-22T03:23:23.047579+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1180736,"data_compressed":59924,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":155,"seq":665719930975,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":40552,"kb_used_data":1496,"kb_used_omap":0,"kb_used_meta":39040,"kb_avail":94331288,"statfs":{"total":96636764160,"available":96595238912,"internally_reserved":0,"allocated":1531904,"data_stored":616310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":39976960},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":159,"seq":682899800134,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":43172,"kb_used_data":1556,"kb_used_omap":0,"kb_used_meta":41600,"kb_avail":94328668,"statfs":{"total":96636764160,"available":96592556032,"internally_reserved":0,"allocated":1593344,"data_stored":1206678,"data_compressed":29962,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":42598400},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":170,"seq":730144440339,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":39972,"kb_used_data":1556,"kb_used_omap":0,"kb_used_meta":38400,"kb_avail":94331868,"statfs":{"total":96636764160,"available":96595832832,"internally_reserved":0,"allocated":1593344,"data_stored":1206678,"data_compressed":29962,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":39321600},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":163,"seq":700079669303,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":37664,"kb_used_data":1552,"kb_used_omap":0,"kb_used_meta":36096,"kb_avail":94334176,"statfs":{"total":96636764160,"available":96598196224,"internally_reserved":0,"allocated":1589248,"data_stored":1206659,"data_compressed":29962,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":36962304},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29962,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29962,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-22T03:23:27.117 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-22T03:23:27.117 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-22T03:23:27.222 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:23:27.217+0000 7f1babd97640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:23:27.225 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:23:27.223+0000 7f1babd97640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:23:27.400 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-22T03:23:27.401 INFO:teuthology.orchestra.run.smithi153.stderr:dumped all 2024-09-22T03:23:27.413 INFO:teuthology.orchestra.run.smithi153.stdout:{"pg_ready":true,"pg_map":{"version":637,"stamp":"2024-09-22T03:23:26.303433+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":139,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":27,"acting":27,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":161360,"kb_used_data":6160,"kb_used_omap":0,"kb_used_meta":155136,"kb_avail":377326000,"statfs":{"total":386547056640,"available":386381824000,"internally_reserved":0,"allocated":6307840,"data_stored":4236325,"data_compressed":89886,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":158859264},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001200"},"pg_stats":[{"pgid":"1.0","version":"0'0","reported_seq":284,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-22T03:23:23.046203+0000","last_change":"2024-09-22T03:22:39.015859+0000","last_active":"2024-09-22T03:23:23.046203+0000","last_peered":"2024-09-22T03:23:23.046203+0000","last_clean":"2024-09-22T03:23:23.046203+0000","last_became_active":"2024-09-22T03:22:30.096861+0000","last_became_peered":"2024-09-22T03:22:30.096861+0000","last_unstale":"2024-09-22T03:23:23.046203+0000","last_undegraded":"2024-09-22T03:23:23.046203+0000","last_fullsized":"2024-09-22T03:23:23.046203+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":419,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-22T03:23:23.048353+0000","last_change":"2024-09-22T03:23:08.987134+0000","last_active":"2024-09-22T03:23:23.048353+0000","last_peered":"2024-09-22T03:23:23.048353+0000","last_clean":"2024-09-22T03:23:23.048353+0000","last_became_active":"2024-09-22T03:23:06.048233+0000","last_became_peered":"2024-09-22T03:23:06.048233+0000","last_unstale":"2024-09-22T03:23:23.048353+0000","last_undegraded":"2024-09-22T03:23:23.048353+0000","last_fullsized":"2024-09-22T03:23:23.048353+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":447,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-22T03:23:23.046737+0000","last_change":"2024-09-22T03:23:08.992662+0000","last_active":"2024-09-22T03:23:23.046737+0000","last_peered":"2024-09-22T03:23:23.046737+0000","last_clean":"2024-09-22T03:23:23.046737+0000","last_became_active":"2024-09-22T03:23:06.069718+0000","last_became_peered":"2024-09-22T03:23:06.069718+0000","last_unstale":"2024-09-22T03:23:23.046737+0000","last_undegraded":"2024-09-22T03:23:23.046737+0000","last_fullsized":"2024-09-22T03:23:23.046737+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":334,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-22T03:23:23.047694+0000","last_change":"2024-09-22T03:23:08.993326+0000","last_active":"2024-09-22T03:23:23.047694+0000","last_peered":"2024-09-22T03:23:23.047694+0000","last_clean":"2024-09-22T03:23:23.047694+0000","last_became_active":"2024-09-22T03:23:06.050619+0000","last_became_peered":"2024-09-22T03:23:06.050619+0000","last_unstale":"2024-09-22T03:23:23.047694+0000","last_undegraded":"2024-09-22T03:23:23.047694+0000","last_fullsized":"2024-09-22T03:23:23.047694+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"12'97","reported_seq":538,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-22T03:23:23.047931+0000","last_change":"2024-09-22T03:23:06.061505+0000","last_active":"2024-09-22T03:23:23.047931+0000","last_peered":"2024-09-22T03:23:23.047931+0000","last_clean":"2024-09-22T03:23:23.047931+0000","last_became_active":"2024-09-22T03:23:06.050659+0000","last_became_peered":"2024-09-22T03:23:06.050659+0000","last_unstale":"2024-09-22T03:23:23.047931+0000","last_undegraded":"2024-09-22T03:23:23.047931+0000","last_fullsized":"2024-09-22T03:23:23.047931+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":365,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-22T03:23:23.047132+0000","last_change":"2024-09-22T03:23:09.008207+0000","last_active":"2024-09-22T03:23:23.047132+0000","last_peered":"2024-09-22T03:23:23.047132+0000","last_clean":"2024-09-22T03:23:23.047132+0000","last_became_active":"2024-09-22T03:23:06.123987+0000","last_became_peered":"2024-09-22T03:23:06.123987+0000","last_unstale":"2024-09-22T03:23:23.047132+0000","last_undegraded":"2024-09-22T03:23:23.047132+0000","last_fullsized":"2024-09-22T03:23:23.047132+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":415,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-22T03:23:23.046206+0000","last_change":"2024-09-22T03:22:39.026321+0000","last_active":"2024-09-22T03:23:23.046206+0000","last_peered":"2024-09-22T03:23:23.046206+0000","last_clean":"2024-09-22T03:23:23.046206+0000","last_became_active":"2024-09-22T03:22:30.055276+0000","last_became_peered":"2024-09-22T03:22:30.055276+0000","last_unstale":"2024-09-22T03:23:23.046206+0000","last_undegraded":"2024-09-22T03:23:23.046206+0000","last_fullsized":"2024-09-22T03:23:23.046206+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":430,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-22T03:23:23.046759+0000","last_change":"2024-09-22T03:23:09.006611+0000","last_active":"2024-09-22T03:23:23.046759+0000","last_peered":"2024-09-22T03:23:23.046759+0000","last_clean":"2024-09-22T03:23:23.046759+0000","last_became_active":"2024-09-22T03:23:06.062711+0000","last_became_peered":"2024-09-22T03:23:06.062711+0000","last_unstale":"2024-09-22T03:23:23.046759+0000","last_undegraded":"2024-09-22T03:23:23.046759+0000","last_fullsized":"2024-09-22T03:23:23.046759+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":450,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-22T03:23:23.047579+0000","last_change":"2024-09-22T03:23:09.006281+0000","last_active":"2024-09-22T03:23:23.047579+0000","last_peered":"2024-09-22T03:23:23.047579+0000","last_clean":"2024-09-22T03:23:23.047579+0000","last_became_active":"2024-09-22T03:23:06.094384+0000","last_became_peered":"2024-09-22T03:23:06.094384+0000","last_unstale":"2024-09-22T03:23:23.047579+0000","last_undegraded":"2024-09-22T03:23:23.047579+0000","last_fullsized":"2024-09-22T03:23:23.047579+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1180736,"data_compressed":59924,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":155,"seq":665719930975,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":40552,"kb_used_data":1496,"kb_used_omap":0,"kb_used_meta":39040,"kb_avail":94331288,"statfs":{"total":96636764160,"available":96595238912,"internally_reserved":0,"allocated":1531904,"data_stored":616310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":39976960},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":159,"seq":682899800134,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":43172,"kb_used_data":1556,"kb_used_omap":0,"kb_used_meta":41600,"kb_avail":94328668,"statfs":{"total":96636764160,"available":96592556032,"internally_reserved":0,"allocated":1593344,"data_stored":1206678,"data_compressed":29962,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":42598400},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":170,"seq":730144440339,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":39972,"kb_used_data":1556,"kb_used_omap":0,"kb_used_meta":38400,"kb_avail":94331868,"statfs":{"total":96636764160,"available":96595832832,"internally_reserved":0,"allocated":1593344,"data_stored":1206678,"data_compressed":29962,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":39321600},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":163,"seq":700079669303,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":37664,"kb_used_data":1552,"kb_used_omap":0,"kb_used_meta":36096,"kb_avail":94334176,"statfs":{"total":96636764160,"available":96598196224,"internally_reserved":0,"allocated":1589248,"data_stored":1206659,"data_compressed":29962,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":36962304},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29962,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29962,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-22T03:23:27.414 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-22T03:23:27.521 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:23:27.517+0000 7fbc7c0d7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:23:27.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:23:27.523+0000 7fbc7c0d7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:23:27.701 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-22T03:23:27.701 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":182,"fsid":"f17abef2-bc08-46d2-abf1-d47a95f0e2ee","created":"2024-09-22T03:07:53.859391+0000","modified":"2024-09-22T03:23:23.037949+0000","last_up_change":"2024-09-22T03:23:04.979224+0000","last_in_change":"2024-09-22T03:07:55.944112+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2024-09-22T03:08:04.049314+0000","flags":270369,"flags_names":"hashpspool,nopgchange,selfmanaged_snaps,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":14,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".mgr","create_time":"2024-09-22T03:08:04.133626+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.75,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"35bc7c89-cb7c-4664-bed6-4088394939d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":142,"last_clean_end":156,"up_from":163,"up_thru":170,"down_at":161,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6803","nonce":4121670064}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":4121670064}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6805","nonce":4121670064}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":4121670064}]},"public_addr":"172.21.15.153:6803/4121670064","cluster_addr":"172.21.15.153:6804/4121670064","heartbeat_back_addr":"172.21.15.153:6805/4121670064","heartbeat_front_addr":"172.21.15.153:6806/4121670064","state":["exists","up"]},{"osd":1,"uuid":"513ab51d-4bac-4f1a-ade1-755e26e4c8e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":138,"last_clean_end":168,"up_from":170,"up_thru":170,"down_at":169,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6807","nonce":1104835390}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":1104835390}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6801","nonce":1104835390}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":1104835390}]},"public_addr":"172.21.15.153:6807/1104835390","cluster_addr":"172.21.15.153:6802/1104835390","heartbeat_back_addr":"172.21.15.153:6801/1104835390","heartbeat_front_addr":"172.21.15.153:6800/1104835390","state":["exists","up"]},{"osd":2,"uuid":"dd57bc3a-b76e-4c7f-8935-6c2b7eb8ef7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":100,"last_clean_end":152,"up_from":159,"up_thru":170,"down_at":157,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6810","nonce":2571484571}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6811","nonce":2571484571}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6813","nonce":2571484571}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6814","nonce":2571484571}]},"public_addr":"172.21.15.196:6810/2571484571","cluster_addr":"172.21.15.196:6811/2571484571","heartbeat_back_addr":"172.21.15.196:6813/2571484571","heartbeat_front_addr":"172.21.15.196:6814/2571484571","state":["exists","up"]},{"osd":3,"uuid":"76e7924f-5812-47ee-a5ee-ec837d12f327","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":146,"last_clean_end":152,"up_from":155,"up_thru":170,"down_at":153,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6803","nonce":126290868}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6804","nonce":126290868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6806","nonce":126290868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6807","nonce":126290868}]},"public_addr":"172.21.15.196:6803/126290868","cluster_addr":"172.21.15.196:6804/126290868","heartbeat_back_addr":"172.21.15.196:6806/126290868","heartbeat_front_addr":"172.21.15.196:6807/126290868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"2024-09-22T03:22:24.973236+0000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"2024-09-22T03:23:03.976941+0000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"2024-09-22T03:21:56.968452+0000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"2024-09-22T03:21:44.981530+0000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":1,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-22T03:23:28.715 INFO:tasks.ceph:Scrubbing osd.0 2024-09-22T03:23:28.715 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-09-22T03:23:28.832 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:23:28.822+0000 7f8c7aa63640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:23:28.835 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:23:28.833+0000 7f8c7aa63640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:23:28.846 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-22T03:23:28.846 INFO:teuthology.orchestra.run.smithi153.stdout: "success": "" 2024-09-22T03:23:28.846 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-22T03:23:28.858 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-09-22T03:23:28.962 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:23:28.956+0000 7f9f062d7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:23:28.965 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:23:28.964+0000 7f9f062d7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:23:29.144 INFO:teuthology.orchestra.run.smithi153.stderr:instructed osd(s) 0 to deep-scrub 2024-09-22T03:23:29.159 INFO:tasks.ceph:Scrubbing osd.1 2024-09-22T03:23:29.159 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-09-22T03:23:29.268 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:23:29.259+0000 7fb850ad7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:23:29.271 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:23:29.269+0000 7fb850ad7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:23:29.283 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-22T03:23:29.283 INFO:teuthology.orchestra.run.smithi153.stdout: "success": "" 2024-09-22T03:23:29.283 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-22T03:23:29.294 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-09-22T03:23:29.400 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:23:29.394+0000 7f3fceb0e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:23:29.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:23:29.401+0000 7f3fceb0e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:23:29.579 INFO:teuthology.orchestra.run.smithi153.stderr:instructed osd(s) 1 to deep-scrub 2024-09-22T03:23:29.596 INFO:tasks.ceph:Scrubbing osd.2 2024-09-22T03:23:29.596 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-09-22T03:23:29.705 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:23:29.696+0000 7fccbb357640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:23:29.708 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:23:29.706+0000 7fccbb357640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:23:29.720 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-22T03:23:29.720 INFO:teuthology.orchestra.run.smithi153.stdout: "success": "" 2024-09-22T03:23:29.720 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-22T03:23:29.731 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-09-22T03:23:29.835 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:23:29.829+0000 7f286906c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:23:29.838 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:23:29.837+0000 7f286906c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:23:30.014 INFO:teuthology.orchestra.run.smithi153.stderr:instructed osd(s) 2 to deep-scrub 2024-09-22T03:23:30.029 INFO:tasks.ceph:Scrubbing osd.3 2024-09-22T03:23:30.029 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-09-22T03:23:30.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:23:30.134+0000 7f07b2cd7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:23:30.147 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:23:30.146+0000 7f07b2cd7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:23:30.160 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-22T03:23:30.160 INFO:teuthology.orchestra.run.smithi153.stdout: "success": "" 2024-09-22T03:23:30.160 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-22T03:23:30.171 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-09-22T03:23:30.276 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:23:30.270+0000 7f1c87997640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:23:30.279 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:23:30.278+0000 7f1c87997640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:23:30.459 INFO:teuthology.orchestra.run.smithi153.stderr:instructed osd(s) 3 to deep-scrub 2024-09-22T03:23:30.476 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-22T03:23:30.581 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:23:30.576+0000 7fb441957640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:23:30.584 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:23:30.583+0000 7fb441957640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:23:30.758 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-22T03:23:30.758 INFO:teuthology.orchestra.run.smithi153.stderr:dumped all 2024-09-22T03:23:30.771 INFO:teuthology.orchestra.run.smithi153.stdout:{"pg_ready":true,"pg_map":{"version":639,"stamp":"2024-09-22T03:23:30.304310+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":139,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":27,"acting":27,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":161360,"kb_used_data":6160,"kb_used_omap":0,"kb_used_meta":155136,"kb_avail":377326000,"statfs":{"total":386547056640,"available":386381824000,"internally_reserved":0,"allocated":6307840,"data_stored":4236325,"data_compressed":89886,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":158859264},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.270044"},"pg_stats":[{"pgid":"1.0","version":"0'0","reported_seq":284,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-22T03:23:23.046203+0000","last_change":"2024-09-22T03:22:39.015859+0000","last_active":"2024-09-22T03:23:23.046203+0000","last_peered":"2024-09-22T03:23:23.046203+0000","last_clean":"2024-09-22T03:23:23.046203+0000","last_became_active":"2024-09-22T03:22:30.096861+0000","last_became_peered":"2024-09-22T03:22:30.096861+0000","last_unstale":"2024-09-22T03:23:23.046203+0000","last_undegraded":"2024-09-22T03:23:23.046203+0000","last_fullsized":"2024-09-22T03:23:23.046203+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":421,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-22T03:23:29.144449+0000","last_change":"2024-09-22T03:23:29.151035+0000","last_active":"2024-09-22T03:23:29.144449+0000","last_peered":"2024-09-22T03:23:29.144449+0000","last_clean":"2024-09-22T03:23:29.144449+0000","last_became_active":"2024-09-22T03:23:06.048233+0000","last_became_peered":"2024-09-22T03:23:06.048233+0000","last_unstale":"2024-09-22T03:23:29.144449+0000","last_undegraded":"2024-09-22T03:23:29.144449+0000","last_fullsized":"2024-09-22T03:23:29.144449+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:23:29.150864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:23:29.150864+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":447,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-22T03:23:23.046737+0000","last_change":"2024-09-22T03:23:08.992662+0000","last_active":"2024-09-22T03:23:23.046737+0000","last_peered":"2024-09-22T03:23:23.046737+0000","last_clean":"2024-09-22T03:23:23.046737+0000","last_became_active":"2024-09-22T03:23:06.069718+0000","last_became_peered":"2024-09-22T03:23:06.069718+0000","last_unstale":"2024-09-22T03:23:23.046737+0000","last_undegraded":"2024-09-22T03:23:23.046737+0000","last_fullsized":"2024-09-22T03:23:23.046737+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":334,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-22T03:23:23.047694+0000","last_change":"2024-09-22T03:23:08.993326+0000","last_active":"2024-09-22T03:23:23.047694+0000","last_peered":"2024-09-22T03:23:23.047694+0000","last_clean":"2024-09-22T03:23:23.047694+0000","last_became_active":"2024-09-22T03:23:06.050619+0000","last_became_peered":"2024-09-22T03:23:06.050619+0000","last_unstale":"2024-09-22T03:23:23.047694+0000","last_undegraded":"2024-09-22T03:23:23.047694+0000","last_fullsized":"2024-09-22T03:23:23.047694+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"12'97","reported_seq":540,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-22T03:23:29.144974+0000","last_change":"2024-09-22T03:23:29.168901+0000","last_active":"2024-09-22T03:23:29.144974+0000","last_peered":"2024-09-22T03:23:29.144974+0000","last_clean":"2024-09-22T03:23:29.144974+0000","last_became_active":"2024-09-22T03:23:06.050659+0000","last_became_peered":"2024-09-22T03:23:06.050659+0000","last_unstale":"2024-09-22T03:23:29.144974+0000","last_undegraded":"2024-09-22T03:23:29.144974+0000","last_fullsized":"2024-09-22T03:23:29.144974+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"12'97","last_scrub_stamp":"2024-09-22T03:23:29.168749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:23:29.168749+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":365,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-22T03:23:23.047132+0000","last_change":"2024-09-22T03:23:09.008207+0000","last_active":"2024-09-22T03:23:23.047132+0000","last_peered":"2024-09-22T03:23:23.047132+0000","last_clean":"2024-09-22T03:23:23.047132+0000","last_became_active":"2024-09-22T03:23:06.123987+0000","last_became_peered":"2024-09-22T03:23:06.123987+0000","last_unstale":"2024-09-22T03:23:23.047132+0000","last_undegraded":"2024-09-22T03:23:23.047132+0000","last_fullsized":"2024-09-22T03:23:23.047132+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":415,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-22T03:23:23.046206+0000","last_change":"2024-09-22T03:22:39.026321+0000","last_active":"2024-09-22T03:23:23.046206+0000","last_peered":"2024-09-22T03:23:23.046206+0000","last_clean":"2024-09-22T03:23:23.046206+0000","last_became_active":"2024-09-22T03:22:30.055276+0000","last_became_peered":"2024-09-22T03:22:30.055276+0000","last_unstale":"2024-09-22T03:23:23.046206+0000","last_undegraded":"2024-09-22T03:23:23.046206+0000","last_fullsized":"2024-09-22T03:23:23.046206+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":430,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-22T03:23:23.046759+0000","last_change":"2024-09-22T03:23:09.006611+0000","last_active":"2024-09-22T03:23:23.046759+0000","last_peered":"2024-09-22T03:23:23.046759+0000","last_clean":"2024-09-22T03:23:23.046759+0000","last_became_active":"2024-09-22T03:23:06.062711+0000","last_became_peered":"2024-09-22T03:23:06.062711+0000","last_unstale":"2024-09-22T03:23:23.046759+0000","last_undegraded":"2024-09-22T03:23:23.046759+0000","last_fullsized":"2024-09-22T03:23:23.046759+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":450,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-22T03:23:23.047579+0000","last_change":"2024-09-22T03:23:09.006281+0000","last_active":"2024-09-22T03:23:23.047579+0000","last_peered":"2024-09-22T03:23:23.047579+0000","last_clean":"2024-09-22T03:23:23.047579+0000","last_became_active":"2024-09-22T03:23:06.094384+0000","last_became_peered":"2024-09-22T03:23:06.094384+0000","last_unstale":"2024-09-22T03:23:23.047579+0000","last_undegraded":"2024-09-22T03:23:23.047579+0000","last_fullsized":"2024-09-22T03:23:23.047579+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:04.133992+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1180736,"data_compressed":59924,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":155,"seq":665719930980,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":40552,"kb_used_data":1496,"kb_used_omap":0,"kb_used_meta":39040,"kb_avail":94331288,"statfs":{"total":96636764160,"available":96595238912,"internally_reserved":0,"allocated":1531904,"data_stored":616310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":39976960},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":159,"seq":682899800134,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":43172,"kb_used_data":1556,"kb_used_omap":0,"kb_used_meta":41600,"kb_avail":94328668,"statfs":{"total":96636764160,"available":96592556032,"internally_reserved":0,"allocated":1593344,"data_stored":1206678,"data_compressed":29962,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":42598400},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":170,"seq":730144440344,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":39972,"kb_used_data":1556,"kb_used_omap":0,"kb_used_meta":38400,"kb_avail":94331868,"statfs":{"total":96636764160,"available":96595832832,"internally_reserved":0,"allocated":1593344,"data_stored":1206678,"data_compressed":29962,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":39321600},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":163,"seq":700079669308,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":37664,"kb_used_data":1552,"kb_used_omap":0,"kb_used_meta":36096,"kb_avail":94334176,"statfs":{"total":96636764160,"available":96598196224,"internally_reserved":0,"allocated":1589248,"data_stored":1206659,"data_compressed":29962,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":36962304},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29962,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29962,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-22T03:23:30.773 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-09-22T03:08:04.133992+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=22, tm_hour=3, tm_min=8, tm_sec=4, tm_wday=6, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=22, tm_hour=3, tm_min=23, tm_sec=27, tm_wday=6, tm_yday=266, tm_isdst=0) 2024-09-22T03:23:30.773 INFO:tasks.ceph:pgid 1.2 last_scrub_stamp 2024-09-22T03:08:04.133992+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=22, tm_hour=3, tm_min=8, tm_sec=4, tm_wday=6, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=22, tm_hour=3, tm_min=23, tm_sec=27, tm_wday=6, tm_yday=266, tm_isdst=0) 2024-09-22T03:23:30.774 INFO:tasks.ceph:pgid 1.3 last_scrub_stamp 2024-09-22T03:08:04.133992+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=22, tm_hour=3, tm_min=8, tm_sec=4, tm_wday=6, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=22, tm_hour=3, tm_min=23, tm_sec=27, tm_wday=6, tm_yday=266, tm_isdst=0) 2024-09-22T03:23:30.774 INFO:tasks.ceph:pgid 1.4 last_scrub_stamp 2024-09-22T03:08:04.133992+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=22, tm_hour=3, tm_min=8, tm_sec=4, tm_wday=6, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=22, tm_hour=3, tm_min=23, tm_sec=27, tm_wday=6, tm_yday=266, tm_isdst=0) 2024-09-22T03:23:30.774 INFO:tasks.ceph:pgid 1.5 last_scrub_stamp 2024-09-22T03:08:04.133992+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=22, tm_hour=3, tm_min=8, tm_sec=4, tm_wday=6, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=22, tm_hour=3, tm_min=23, tm_sec=27, tm_wday=6, tm_yday=266, tm_isdst=0) 2024-09-22T03:23:30.774 INFO:tasks.ceph:pgid 1.6 last_scrub_stamp 2024-09-22T03:08:04.133992+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=22, tm_hour=3, tm_min=8, tm_sec=4, tm_wday=6, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=22, tm_hour=3, tm_min=23, tm_sec=27, tm_wday=6, tm_yday=266, tm_isdst=0) 2024-09-22T03:23:30.774 INFO:tasks.ceph:pgid 1.7 last_scrub_stamp 2024-09-22T03:08:04.133992+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=22, tm_hour=3, tm_min=8, tm_sec=4, tm_wday=6, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=22, tm_hour=3, tm_min=23, tm_sec=27, tm_wday=6, tm_yday=266, tm_isdst=0) 2024-09-22T03:23:30.774 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-09-22T03:23:50.775 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-22T03:23:50.892 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:23:50.887+0000 7fb7048d7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:23:50.895 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:23:50.893+0000 7fb7048d7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:23:51.074 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-22T03:23:51.074 INFO:teuthology.orchestra.run.smithi153.stderr:dumped all 2024-09-22T03:23:51.087 INFO:teuthology.orchestra.run.smithi153.stdout:{"pg_ready":true,"pg_map":{"version":649,"stamp":"2024-09-22T03:23:50.309247+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":139,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":27,"acting":27,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":161360,"kb_used_data":6160,"kb_used_omap":0,"kb_used_meta":155136,"kb_avail":377326000,"statfs":{"total":386547056640,"available":386381824000,"internally_reserved":0,"allocated":6307840,"data_stored":4236325,"data_compressed":89886,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":158859264},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002889"},"pg_stats":[{"pgid":"1.0","version":"0'0","reported_seq":286,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-22T03:23:30.460263+0000","last_change":"2024-09-22T03:23:30.466417+0000","last_active":"2024-09-22T03:23:30.460263+0000","last_peered":"2024-09-22T03:23:30.460263+0000","last_clean":"2024-09-22T03:23:30.460263+0000","last_became_active":"2024-09-22T03:22:30.096861+0000","last_became_peered":"2024-09-22T03:22:30.096861+0000","last_unstale":"2024-09-22T03:23:30.460263+0000","last_undegraded":"2024-09-22T03:23:30.460263+0000","last_fullsized":"2024-09-22T03:23:30.460263+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:23:30.466283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:23:30.466283+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":421,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-22T03:23:29.144449+0000","last_change":"2024-09-22T03:23:29.151035+0000","last_active":"2024-09-22T03:23:29.144449+0000","last_peered":"2024-09-22T03:23:29.144449+0000","last_clean":"2024-09-22T03:23:29.144449+0000","last_became_active":"2024-09-22T03:23:06.048233+0000","last_became_peered":"2024-09-22T03:23:06.048233+0000","last_unstale":"2024-09-22T03:23:29.144449+0000","last_undegraded":"2024-09-22T03:23:29.144449+0000","last_fullsized":"2024-09-22T03:23:29.144449+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:23:29.150864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:23:29.150864+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":449,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-22T03:23:30.014842+0000","last_change":"2024-09-22T03:23:30.024450+0000","last_active":"2024-09-22T03:23:30.014842+0000","last_peered":"2024-09-22T03:23:30.014842+0000","last_clean":"2024-09-22T03:23:30.014842+0000","last_became_active":"2024-09-22T03:23:06.069718+0000","last_became_peered":"2024-09-22T03:23:06.069718+0000","last_unstale":"2024-09-22T03:23:30.014842+0000","last_undegraded":"2024-09-22T03:23:30.014842+0000","last_fullsized":"2024-09-22T03:23:30.014842+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"14'2","last_scrub_stamp":"2024-09-22T03:23:30.024289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:23:30.024289+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":336,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-22T03:23:30.460266+0000","last_change":"2024-09-22T03:23:30.467851+0000","last_active":"2024-09-22T03:23:30.460266+0000","last_peered":"2024-09-22T03:23:30.460266+0000","last_clean":"2024-09-22T03:23:30.460266+0000","last_became_active":"2024-09-22T03:23:06.050619+0000","last_became_peered":"2024-09-22T03:23:06.050619+0000","last_unstale":"2024-09-22T03:23:30.460266+0000","last_undegraded":"2024-09-22T03:23:30.460266+0000","last_fullsized":"2024-09-22T03:23:30.460266+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"12'1","last_scrub_stamp":"2024-09-22T03:23:30.467733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:23:30.467733+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"12'97","reported_seq":540,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-22T03:23:29.144974+0000","last_change":"2024-09-22T03:23:29.168901+0000","last_active":"2024-09-22T03:23:29.144974+0000","last_peered":"2024-09-22T03:23:29.144974+0000","last_clean":"2024-09-22T03:23:29.144974+0000","last_became_active":"2024-09-22T03:23:06.050659+0000","last_became_peered":"2024-09-22T03:23:06.050659+0000","last_unstale":"2024-09-22T03:23:29.144974+0000","last_undegraded":"2024-09-22T03:23:29.144974+0000","last_fullsized":"2024-09-22T03:23:29.144974+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"12'97","last_scrub_stamp":"2024-09-22T03:23:29.168749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:23:29.168749+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":367,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-22T03:23:29.580518+0000","last_change":"2024-09-22T03:23:29.586696+0000","last_active":"2024-09-22T03:23:29.580518+0000","last_peered":"2024-09-22T03:23:29.580518+0000","last_clean":"2024-09-22T03:23:29.580518+0000","last_became_active":"2024-09-22T03:23:06.123987+0000","last_became_peered":"2024-09-22T03:23:06.123987+0000","last_unstale":"2024-09-22T03:23:29.580518+0000","last_undegraded":"2024-09-22T03:23:29.580518+0000","last_fullsized":"2024-09-22T03:23:29.580518+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:23:29.586588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:23:29.586588+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":417,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-22T03:23:30.015363+0000","last_change":"2024-09-22T03:23:30.021533+0000","last_active":"2024-09-22T03:23:30.015363+0000","last_peered":"2024-09-22T03:23:30.015363+0000","last_clean":"2024-09-22T03:23:30.015363+0000","last_became_active":"2024-09-22T03:22:30.055276+0000","last_became_peered":"2024-09-22T03:22:30.055276+0000","last_unstale":"2024-09-22T03:23:30.015363+0000","last_undegraded":"2024-09-22T03:23:30.015363+0000","last_fullsized":"2024-09-22T03:23:30.015363+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:23:30.021403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:23:30.021403+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":432,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-22T03:23:29.580445+0000","last_change":"2024-09-22T03:23:29.588520+0000","last_active":"2024-09-22T03:23:29.580445+0000","last_peered":"2024-09-22T03:23:29.580445+0000","last_clean":"2024-09-22T03:23:29.580445+0000","last_became_active":"2024-09-22T03:23:06.062711+0000","last_became_peered":"2024-09-22T03:23:06.062711+0000","last_unstale":"2024-09-22T03:23:29.580445+0000","last_undegraded":"2024-09-22T03:23:29.580445+0000","last_fullsized":"2024-09-22T03:23:29.580445+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:23:29.588348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:23:29.588348+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":452,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-22T03:23:29.580943+0000","last_change":"2024-09-22T03:23:29.587271+0000","last_active":"2024-09-22T03:23:29.580943+0000","last_peered":"2024-09-22T03:23:29.580943+0000","last_clean":"2024-09-22T03:23:29.580943+0000","last_became_active":"2024-09-22T03:23:06.094384+0000","last_became_peered":"2024-09-22T03:23:06.094384+0000","last_unstale":"2024-09-22T03:23:29.580943+0000","last_undegraded":"2024-09-22T03:23:29.580943+0000","last_fullsized":"2024-09-22T03:23:29.580943+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:23:29.587105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:23:29.587105+0000","last_clean_scrub_stamp":"2024-09-22T03:08:04.133992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1180736,"data_compressed":59924,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":155,"seq":665719931000,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":40552,"kb_used_data":1496,"kb_used_omap":0,"kb_used_meta":39040,"kb_avail":94331288,"statfs":{"total":96636764160,"available":96595238912,"internally_reserved":0,"allocated":1531904,"data_stored":616310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":39976960},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":159,"seq":682899800154,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":43172,"kb_used_data":1556,"kb_used_omap":0,"kb_used_meta":41600,"kb_avail":94328668,"statfs":{"total":96636764160,"available":96592556032,"internally_reserved":0,"allocated":1593344,"data_stored":1206678,"data_compressed":29962,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":42598400},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":170,"seq":730144440364,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":39972,"kb_used_data":1556,"kb_used_omap":0,"kb_used_meta":38400,"kb_avail":94331868,"statfs":{"total":96636764160,"available":96595832832,"internally_reserved":0,"allocated":1593344,"data_stored":1206678,"data_compressed":29962,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":39321600},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":163,"seq":700079669328,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":37664,"kb_used_data":1552,"kb_used_omap":0,"kb_used_meta":36096,"kb_avail":94334176,"statfs":{"total":96636764160,"available":96598196224,"internally_reserved":0,"allocated":1589248,"data_stored":1206659,"data_compressed":29962,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":36962304},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29962,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29962,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-22T03:23:51.088 DEBUG:teuthology.orchestra.run.smithi153:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-09-22T03:23:51.193 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:23:51.182+0000 7fa349397640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:23:51.194 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-22T03:23:51.192+0000 7fa349397640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:23:51.388 INFO:teuthology.misc:Shutting down mds daemons... 2024-09-22T03:23:51.388 INFO:teuthology.misc:Shutting down osd daemons... 2024-09-22T03:23:51.388 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-09-22T03:23:51.388 INFO:teuthology.orchestra.run:waiting for 300 2024-09-22T03:23:55.149 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~0s 2024-09-22T03:23:57.490 INFO:tasks.ceph.osd.0:Stopped 2024-09-22T03:23:57.491 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-09-22T03:23:57.491 INFO:teuthology.orchestra.run:waiting for 300 2024-09-22T03:24:00.553 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~0s 2024-09-22T03:24:00.553 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 has been restored 2024-09-22T03:24:03.592 INFO:tasks.ceph.osd.1:Stopped 2024-09-22T03:24:03.592 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-09-22T03:24:03.592 INFO:teuthology.orchestra.run:waiting for 300 2024-09-22T03:24:05.856 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~0s 2024-09-22T03:24:05.856 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 has been restored 2024-09-22T03:24:09.694 INFO:tasks.ceph.osd.2:Stopped 2024-09-22T03:24:09.695 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-09-22T03:24:09.695 INFO:teuthology.orchestra.run:waiting for 300 2024-09-22T03:24:11.058 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~0s 2024-09-22T03:24:11.058 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 has been restored 2024-09-22T03:24:15.797 INFO:tasks.ceph.osd.3:Stopped 2024-09-22T03:24:15.797 INFO:teuthology.misc:Shutting down mgr daemons... 2024-09-22T03:24:15.797 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-09-22T03:24:15.797 INFO:teuthology.orchestra.run:waiting for 300 2024-09-22T03:24:15.829 INFO:tasks.ceph.mgr.x:Stopped 2024-09-22T03:24:15.829 INFO:teuthology.misc:Shutting down mon daemons... 2024-09-22T03:24:15.829 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-22T03:24:15.829 INFO:teuthology.orchestra.run:waiting for 300 2024-09-22T03:24:15.844 INFO:tasks.ceph.mon.a:Stopped 2024-09-22T03:24:15.844 INFO:tasks.ceph:Checking cluster log for badness... 2024-09-22T03:24:15.845 DEBUG:teuthology.orchestra.run.smithi153:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'timeout on replica' | egrep -v 'late reservation from' | egrep -v MON_DOWN | egrep -v OSDMAP_FLAGS | egrep -v OSD_DOWN | egrep -v PG_DEGRADED | egrep -v PG_AVAILABILITY | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'mons down' | egrep -v 'mon down' | egrep -v 'out of quorum' | egrep -v noscrub | egrep -v nodeep-scrub | head -n 1 2024-09-22T03:24:16.044 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi153.front.sepia.ceph.com 2024-09-22T03:24:16.045 DEBUG:teuthology.orchestra.run.smithi153:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-09-22T03:24:16.059 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 has been restored 2024-09-22T03:24:16.133 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi153.front.sepia.ceph.com 2024-09-22T03:24:16.134 DEBUG:teuthology.orchestra.run.smithi153:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-09-22T03:24:16.196 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi196.front.sepia.ceph.com 2024-09-22T03:24:16.196 DEBUG:teuthology.orchestra.run.smithi196:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-09-22T03:24:16.479 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi196.front.sepia.ceph.com 2024-09-22T03:24:16.479 DEBUG:teuthology.orchestra.run.smithi196:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-09-22T03:24:16.548 INFO:tasks.ceph:Archiving mon data... 2024-09-22T03:24:16.549 DEBUG:teuthology.misc:Transferring archived files from smithi153:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/teuthology-2024-09-21_21:56:02-crimson-rados-squid-distro-crimson-smithi/7914725/data/mon.a.tgz 2024-09-22T03:24:16.549 DEBUG:teuthology.orchestra.run.smithi153:> mktemp 2024-09-22T03:24:16.566 INFO:teuthology.orchestra.run.smithi153.stdout:/tmp/tmp.hDAlwoKEWm 2024-09-22T03:24:16.566 DEBUG:teuthology.orchestra.run.smithi153:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.hDAlwoKEWm 2024-09-22T03:24:16.700 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod 0666 /tmp/tmp.hDAlwoKEWm 2024-09-22T03:24:16.784 DEBUG:teuthology.orchestra.remote:smithi153:/tmp/tmp.hDAlwoKEWm is 303KB 2024-09-22T03:24:16.879 DEBUG:teuthology.orchestra.run.smithi153:> rm -fr /tmp/tmp.hDAlwoKEWm 2024-09-22T03:24:16.894 INFO:tasks.ceph:Cleaning ceph cluster... 2024-09-22T03:24:16.894 DEBUG:teuthology.orchestra.run.smithi153:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-22T03:24:16.936 DEBUG:teuthology.orchestra.run.smithi196:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-22T03:24:17.015 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-09-22T03:24:17.033 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-09-22T03:24:17.033 INFO:tasks.ceph:Archiving crash dumps... 2024-09-22T03:24:17.034 DEBUG:teuthology.misc:Transferring archived files from smithi153:/var/lib/ceph/crash to /home/teuthworker/archive/teuthology-2024-09-21_21:56:02-crimson-rados-squid-distro-crimson-smithi/7914725/remote/smithi153/crash 2024-09-22T03:24:17.035 DEBUG:teuthology.orchestra.run.smithi153:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-22T03:24:17.080 DEBUG:teuthology.misc:Transferring archived files from smithi196:/var/lib/ceph/crash to /home/teuthworker/archive/teuthology-2024-09-21_21:56:02-crimson-rados-squid-distro-crimson-smithi/7914725/remote/smithi196/crash 2024-09-22T03:24:17.081 DEBUG:teuthology.orchestra.run.smithi196:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-22T03:24:17.119 INFO:tasks.ceph:Compressing logs... 2024-09-22T03:24:17.119 DEBUG:teuthology.orchestra.run.smithi153:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-22T03:24:17.126 DEBUG:teuthology.orchestra.run.smithi196:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-22T03:24:17.196 INFO:teuthology.orchestra.run.smithi196.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28212.log 2024-09-22T03:24:17.196 INFO:teuthology.orchestra.run.smithi196.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28252.log 2024-09-22T03:24:17.196 INFO:teuthology.orchestra.run.smithi196.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28121.log 2024-09-22T03:24:17.197 INFO:teuthology.orchestra.run.smithi196.stderr:/var/log/ceph/ceph-client.admin.28212.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.28212.log.gz 2024-09-22T03:24:17.197 INFO:teuthology.orchestra.run.smithi196.stderr:/var/log/ceph/ceph-client.admin.28252.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28172.log 2024-09-22T03:24:17.197 INFO:teuthology.orchestra.run.smithi196.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.28252.log.gz 2024-09-22T03:24:17.197 INFO:teuthology.orchestra.run.smithi196.stderr:/var/log/ceph/ceph-client.admin.28121.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-09-22T03:24:17.197 INFO:teuthology.orchestra.run.smithi196.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.28121.log.gz 2024-09-22T03:24:17.197 INFO:teuthology.orchestra.run.smithi196.stderr:/var/log/ceph/ceph-client.admin.28172.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-09-22T03:24:17.197 INFO:teuthology.orchestra.run.smithi196.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.28172.log.gz 2024-09-22T03:24:17.197 INFO:teuthology.orchestra.run.smithi196.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-09-22T03:24:17.262 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-09-22T03:24:17.263 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37621.log 2024-09-22T03:24:17.263 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49250.log 2024-09-22T03:24:17.263 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-osd.0.log: /var/log/ceph/ceph-client.admin.37621.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31086.log 2024-09-22T03:24:17.263 INFO:teuthology.orchestra.run.smithi153.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.37621.log.gz 2024-09-22T03:24:17.263 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29271.log 2024-09-22T03:24:17.263 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.49250.log: 50.8% -- replaced with /var/log/ceph/ceph-client.admin.49250.log.gz 2024-09-22T03:24:17.263 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.31086.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44163.log 2024-09-22T03:24:17.263 INFO:teuthology.orchestra.run.smithi153.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31086.log.gz 2024-09-22T03:24:17.263 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.29271.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43039.log 2024-09-22T03:24:17.263 INFO:teuthology.orchestra.run.smithi153.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29271.log.gz 2024-09-22T03:24:17.263 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.44163.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37935.log 2024-09-22T03:24:17.263 INFO:teuthology.orchestra.run.smithi153.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.44163.log.gz 2024-09-22T03:24:17.263 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.43039.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40210.log 2024-09-22T03:24:17.264 INFO:teuthology.orchestra.run.smithi153.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.43039.log.gz 2024-09-22T03:24:17.264 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.37935.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45408.log 2024-09-22T03:24:17.264 INFO:teuthology.orchestra.run.smithi153.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.37935.log.gz 2024-09-22T03:24:17.264 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.40210.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29170.log 2024-09-22T03:24:17.264 INFO:teuthology.orchestra.run.smithi153.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.40210.log.gz 2024-09-22T03:24:17.264 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.45408.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39253.log 2024-09-22T03:24:17.264 INFO:teuthology.orchestra.run.smithi153.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.45408.log.gz 2024-09-22T03:24:17.264 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.29170.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.43901.log 2024-09-22T03:24:17.264 INFO:teuthology.orchestra.run.smithi153.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29170.log.gz 2024-09-22T03:24:17.264 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.39253.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47458.log 2024-09-22T03:24:17.264 INFO:teuthology.orchestra.run.smithi153.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.39253.log.gz 2024-09-22T03:24:17.264 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.0.43901.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47625.log 2024-09-22T03:24:17.264 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.47458.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47758.log 2024-09-22T03:24:17.264 INFO:teuthology.orchestra.run.smithi153.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.47458.log.gz 2024-09-22T03:24:17.264 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.47625.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.47625.log.gz 2024-09-22T03:24:17.265 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37982.log 2024-09-22T03:24:17.265 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.47758.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34816.log 2024-09-22T03:24:17.265 INFO:teuthology.orchestra.run.smithi153.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.47758.log.gz 2024-09-22T03:24:17.265 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.37982.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.37982.log.gz 2024-09-22T03:24:17.265 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45359.log 2024-09-22T03:24:17.265 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.34816.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33924.log 2024-09-22T03:24:17.265 INFO:teuthology.orchestra.run.smithi153.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.34816.log.gz 2024-09-22T03:24:17.265 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.45359.log: 50.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.45359.log.gz -5 2024-09-22T03:24:17.265 INFO:teuthology.orchestra.run.smithi153.stderr: --verbose -- /var/log/ceph/ceph-client.admin.47799.log 2024-09-22T03:24:17.265 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33249.log 2024-09-22T03:24:17.265 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.33924.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.33924.log.gz 2024-09-22T03:24:17.265 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.47799.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.47799.log.gz 2024-09-22T03:24:17.265 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36228.log 2024-09-22T03:24:17.266 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45855.log 2024-09-22T03:24:17.266 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.33249.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.33249.log.gz 2024-09-22T03:24:17.266 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.36228.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48684.log 2024-09-22T03:24:17.266 INFO:teuthology.orchestra.run.smithi153.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.36228.log.gz 2024-09-22T03:24:17.266 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.45855.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38928.log 2024-09-22T03:24:17.266 INFO:teuthology.orchestra.run.smithi153.stderr: 47.6% -- replaced with /var/log/ceph/ceph-client.admin.45855.log.gz 2024-09-22T03:24:17.266 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.48684.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.48684.log.gz 2024-09-22T03:24:17.266 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48810.log 2024-09-22T03:24:17.266 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.38928.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29716.log 2024-09-22T03:24:17.266 INFO:teuthology.orchestra.run.smithi153.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.38928.log.gz 2024-09-22T03:24:17.266 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.48810.log: gzip 50.8% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.48810.log.gz -- 2024-09-22T03:24:17.266 INFO:teuthology.orchestra.run.smithi153.stderr: /var/log/ceph/ceph-client.admin.32185.log 2024-09-22T03:24:17.266 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.29716.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31828.log 2024-09-22T03:24:17.267 INFO:teuthology.orchestra.run.smithi153.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29716.log.gz 2024-09-22T03:24:17.267 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.32185.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.32185.log.gz 2024-09-22T03:24:17.267 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46283.log 2024-09-22T03:24:17.267 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.31828.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35304.log 2024-09-22T03:24:17.267 INFO:teuthology.orchestra.run.smithi153.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31828.log.gz 2024-09-22T03:24:17.267 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.46283.log: gzip 49.6% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.46283.log.gz 2024-09-22T03:24:17.267 INFO:teuthology.orchestra.run.smithi153.stderr: -- /var/log/ceph/ceph-client.admin.49650.log 2024-09-22T03:24:17.267 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.35304.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31643.log 2024-09-22T03:24:17.267 INFO:teuthology.orchestra.run.smithi153.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.35304.log.gz 2024-09-22T03:24:17.267 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.49650.log: 50.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.49650.log.gz 2024-09-22T03:24:17.267 INFO:teuthology.orchestra.run.smithi153.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.35890.log 2024-09-22T03:24:17.267 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40984.log 2024-09-22T03:24:17.267 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.31643.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31643.log.gz 2024-09-22T03:24:17.267 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.35890.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.35890.log.gz 2024-09-22T03:24:17.268 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29674.log 2024-09-22T03:24:17.268 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37664.log 2024-09-22T03:24:17.268 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.40984.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.40984.log.gz 2024-09-22T03:24:17.268 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.29674.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48951.log 2024-09-22T03:24:17.268 INFO:teuthology.orchestra.run.smithi153.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29674.log.gz 2024-09-22T03:24:17.268 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.37664.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38269.log 2024-09-22T03:24:17.268 INFO:teuthology.orchestra.run.smithi153.stderr: 49.2% -- replaced with /var/log/ceph/ceph-client.admin.37664.log.gz 2024-09-22T03:24:17.268 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.48951.log: gzip 50.8% -- replaced with /var/log/ceph/ceph-client.admin.48951.log.gz -5 2024-09-22T03:24:17.268 INFO:teuthology.orchestra.run.smithi153.stderr: --verbose -- /var/log/ceph/ceph-client.admin.43453.log 2024-09-22T03:24:17.268 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.38269.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43088.log 2024-09-22T03:24:17.268 INFO:teuthology.orchestra.run.smithi153.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.38269.log.gz 2024-09-22T03:24:17.268 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.43453.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.43453.log.gz 2024-09-22T03:24:17.268 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42487.log 2024-09-22T03:24:17.268 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.43088.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46469.log 2024-09-22T03:24:17.269 INFO:teuthology.orchestra.run.smithi153.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.43088.log.gz 2024-09-22T03:24:17.269 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.42487.log: gzip -5 --verbose -- 50.4% /var/log/ceph/ceph-client.admin.35436.log 2024-09-22T03:24:17.269 INFO:teuthology.orchestra.run.smithi153.stderr: -- replaced with /var/log/ceph/ceph-client.admin.42487.log.gz 2024-09-22T03:24:17.269 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.46469.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33588.log 2024-09-22T03:24:17.269 INFO:teuthology.orchestra.run.smithi153.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.46469.log.gz 2024-09-22T03:24:17.269 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.35436.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.35436.log.gz 2024-09-22T03:24:17.269 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47409.log 2024-09-22T03:24:17.269 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.33588.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35621.log 2024-09-22T03:24:17.269 INFO:teuthology.orchestra.run.smithi153.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.33588.log.gz 2024-09-22T03:24:17.269 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.47409.log: 49.6%gzip -- replaced with /var/log/ceph/ceph-client.admin.47409.log.gz 2024-09-22T03:24:17.269 INFO:teuthology.orchestra.run.smithi153.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.46843.log 2024-09-22T03:24:17.269 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.35621.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36095.log 2024-09-22T03:24:17.269 INFO:teuthology.orchestra.run.smithi153.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.35621.log.gz 2024-09-22T03:24:17.269 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.46843.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.46843.log.gz 2024-09-22T03:24:17.270 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46884.log 2024-09-22T03:24:17.270 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.36095.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39519.log 2024-09-22T03:24:17.270 INFO:teuthology.orchestra.run.smithi153.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.36095.log.gz 2024-09-22T03:24:17.270 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.46884.log: 49.2% -- replaced with /var/log/ceph/ceph-client.admin.46884.log.gz 2024-09-22T03:24:17.270 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30828.log 2024-09-22T03:24:17.270 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.39519.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43406.log 2024-09-22T03:24:17.270 INFO:teuthology.orchestra.run.smithi153.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.39519.log.gz 2024-09-22T03:24:17.270 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.30828.log: gzip 50.0% -5 -- replaced with /var/log/ceph/ceph-client.admin.30828.log.gz --verbose 2024-09-22T03:24:17.270 INFO:teuthology.orchestra.run.smithi153.stderr: -- /var/log/ceph/ceph-client.admin.41118.log 2024-09-22T03:24:17.270 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.43406.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48381.log 2024-09-22T03:24:17.270 INFO:teuthology.orchestra.run.smithi153.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.43406.log.gz 2024-09-22T03:24:17.270 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.41118.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31910.log 2024-09-22T03:24:17.270 INFO:teuthology.orchestra.run.smithi153.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.41118.log.gz 2024-09-22T03:24:17.270 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.48381.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38318.log 2024-09-22T03:24:17.271 INFO:teuthology.orchestra.run.smithi153.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.48381.log.gz 2024-09-22T03:24:17.271 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.31910.log: gzip 49.6% -5 -- replaced with /var/log/ceph/ceph-client.admin.31910.log.gz --verbose 2024-09-22T03:24:17.271 INFO:teuthology.orchestra.run.smithi153.stderr: -- /var/log/ceph/ceph-client.admin.41753.log 2024-09-22T03:24:17.271 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.38318.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-09-22T03:24:17.271 INFO:teuthology.orchestra.run.smithi153.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.38318.log.gz 2024-09-22T03:24:17.271 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.41753.log: 50.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.41753.log.gz 2024-09-22T03:24:17.271 INFO:teuthology.orchestra.run.smithi153.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.39211.log 2024-09-22T03:24:17.271 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36182.log 2024-09-22T03:24:17.271 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.39211.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.39211.log.gz 2024-09-22T03:24:17.271 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45973.log 2024-09-22T03:24:17.271 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47933.log 2024-09-22T03:24:17.271 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.36182.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.36182.log.gz 2024-09-22T03:24:17.271 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.45973.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.45973.log.gz 2024-09-22T03:24:17.272 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47110.log 2024-09-22T03:24:17.272 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29270.log 2024-09-22T03:24:17.272 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.47933.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.47933.log.gz 2024-09-22T03:24:17.272 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.47110.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.47110.log.gz 2024-09-22T03:24:17.272 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39162.log 2024-09-22T03:24:17.272 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41078.log 2024-09-22T03:24:17.272 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.29270.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29270.log.gz 2024-09-22T03:24:17.272 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.39162.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.39162.log.gz 2024-09-22T03:24:17.272 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47846.log 2024-09-22T03:24:17.272 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49120.log 2024-09-22T03:24:17.272 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.41078.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.41078.log.gz 2024-09-22T03:24:17.272 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.47846.log: gzip 50.0% -- replaced with /var/log/ceph/ceph-client.admin.47846.log.gz 2024-09-22T03:24:17.272 INFO:teuthology.orchestra.run.smithi153.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.49230.log 2024-09-22T03:24:17.272 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.49120.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30437.log 2024-09-22T03:24:17.273 INFO:teuthology.orchestra.run.smithi153.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.49120.log.gz 2024-09-22T03:24:17.273 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.49230.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32026.log 2024-09-22T03:24:17.273 INFO:teuthology.orchestra.run.smithi153.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.49230.log.gz 2024-09-22T03:24:17.273 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.30437.log: gzip -5 --verbose -- 49.6% -- replaced with /var/log/ceph/ceph-client.admin.30437.log.gz 2024-09-22T03:24:17.273 INFO:teuthology.orchestra.run.smithi153.stderr: /var/log/ceph/ceph-client.admin.37154.log 2024-09-22T03:24:17.273 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.32026.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.32026.log.gz 2024-09-22T03:24:17.273 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38520.log 2024-09-22T03:24:17.273 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45084.log 2024-09-22T03:24:17.273 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.37154.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.37154.log.gz 2024-09-22T03:24:17.273 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.38520.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.38520.log.gz 2024-09-22T03:24:17.273 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40557.log 2024-09-22T03:24:17.273 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43109.log 2024-09-22T03:24:17.273 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.45084.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.45084.log.gz 2024-09-22T03:24:17.273 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.40557.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40344.log 2024-09-22T03:24:17.273 INFO:teuthology.orchestra.run.smithi153.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.40557.log.gz 2024-09-22T03:24:17.273 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.43109.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33802.log 2024-09-22T03:24:17.274 INFO:teuthology.orchestra.run.smithi153.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.43109.log.gz 2024-09-22T03:24:17.274 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.40344.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35091.log 2024-09-22T03:24:17.274 INFO:teuthology.orchestra.run.smithi153.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.40344.log.gz 2024-09-22T03:24:17.274 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.33802.log: gzip 49.6% -- replaced with /var/log/ceph/ceph-client.admin.33802.log.gz -5 2024-09-22T03:24:17.274 INFO:teuthology.orchestra.run.smithi153.stderr: --verbose -- /var/log/ceph/ceph-client.admin.39770.log 2024-09-22T03:24:17.274 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.35091.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29884.log 2024-09-22T03:24:17.274 INFO:teuthology.orchestra.run.smithi153.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.35091.log.gz 2024-09-22T03:24:17.274 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.39770.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30044.log 2024-09-22T03:24:17.274 INFO:teuthology.orchestra.run.smithi153.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.39770.log.gz 2024-09-22T03:24:17.274 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.29884.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45755.log 2024-09-22T03:24:17.274 INFO:teuthology.orchestra.run.smithi153.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.29884.log.gz 2024-09-22T03:24:17.274 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.30044.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45835.log 2024-09-22T03:24:17.274 INFO:teuthology.orchestra.run.smithi153.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.30044.log.gz 2024-09-22T03:24:17.274 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35042.log 2024-09-22T03:24:17.275 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.45755.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.45755.log.gz 2024-09-22T03:24:17.275 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.45835.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29548.log 2024-09-22T03:24:17.275 INFO:teuthology.orchestra.run.smithi153.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.45835.log.gz 2024-09-22T03:24:17.275 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.35042.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46567.log 2024-09-22T03:24:17.275 INFO:teuthology.orchestra.run.smithi153.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.35042.log.gz 2024-09-22T03:24:17.275 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.29548.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30948.log 2024-09-22T03:24:17.275 INFO:teuthology.orchestra.run.smithi153.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29548.log.gz 2024-09-22T03:24:17.275 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34509.log 2024-09-22T03:24:17.275 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.46567.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.46567.log.gz 2024-09-22T03:24:17.275 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.30948.log: gzip 50.0% -5 -- replaced with /var/log/ceph/ceph-client.admin.30948.log.gz 2024-09-22T03:24:17.275 INFO:teuthology.orchestra.run.smithi153.stderr: --verbose -- /var/log/ceph/ceph-client.admin.40904.log 2024-09-22T03:24:17.275 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37755.log 2024-09-22T03:24:17.275 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.34509.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.34509.log.gz 2024-09-22T03:24:17.275 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.40904.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38591.log 2024-09-22T03:24:17.275 INFO:teuthology.orchestra.run.smithi153.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.40904.log.gz 2024-09-22T03:24:17.276 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.37755.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39344.log 2024-09-22T03:24:17.276 INFO:teuthology.orchestra.run.smithi153.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.37755.log.gz 2024-09-22T03:24:17.276 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.38591.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34191.log 2024-09-22T03:24:17.276 INFO:teuthology.orchestra.run.smithi153.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.38591.log.gz 2024-09-22T03:24:17.276 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.39344.log: 49.2% -- replaced with /var/log/ceph/ceph-client.admin.39344.log.gz 2024-09-22T03:24:17.276 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37372.log 2024-09-22T03:24:17.276 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.34191.log: gzip 49.6% -- replaced with /var/log/ceph/ceph-client.admin.34191.log.gz -5 2024-09-22T03:24:17.276 INFO:teuthology.orchestra.run.smithi153.stderr: --verbose -- /var/log/ceph/ceph-client.admin.42435.log 2024-09-22T03:24:17.276 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.37372.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.37372.log.gz 2024-09-22T03:24:17.276 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36767.log 2024-09-22T03:24:17.276 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48340.log 2024-09-22T03:24:17.276 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.42435.log: 49.2% -- replaced with /var/log/ceph/ceph-client.admin.42435.log.gz 2024-09-22T03:24:17.276 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.36767.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.36767.log.gz 2024-09-22T03:24:17.276 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45924.log 2024-09-22T03:24:17.277 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41198.log 2024-09-22T03:24:17.277 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.48340.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.48340.log.gz 2024-09-22T03:24:17.277 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.45924.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34013.log 2024-09-22T03:24:17.277 INFO:teuthology.orchestra.run.smithi153.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.45924.log.gz 2024-09-22T03:24:17.277 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.41198.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47157.log 2024-09-22T03:24:17.277 INFO:teuthology.orchestra.run.smithi153.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.41198.log.gz 2024-09-22T03:24:17.277 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.34013.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36013.log 2024-09-22T03:24:17.277 INFO:teuthology.orchestra.run.smithi153.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.34013.log.gz 2024-09-22T03:24:17.277 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.47157.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32006.log 2024-09-22T03:24:17.277 INFO:teuthology.orchestra.run.smithi153.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.47157.log.gz 2024-09-22T03:24:17.277 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.36013.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40384.log 2024-09-22T03:24:17.277 INFO:teuthology.orchestra.run.smithi153.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.36013.log.gz 2024-09-22T03:24:17.277 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.32006.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29269.log 2024-09-22T03:24:17.277 INFO:teuthology.orchestra.run.smithi153.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.32006.log.gz 2024-09-22T03:24:17.277 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.40384.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48474.log 2024-09-22T03:24:17.278 INFO:teuthology.orchestra.run.smithi153.stderr: 49.2% -- replaced with /var/log/ceph/ceph-client.admin.40384.log.gz 2024-09-22T03:24:17.278 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.29269.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35756.log 2024-09-22T03:24:17.278 INFO:teuthology.orchestra.run.smithi153.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.29269.log.gz 2024-09-22T03:24:17.278 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.48474.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36506.log 2024-09-22T03:24:17.278 INFO:teuthology.orchestra.run.smithi153.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.48474.log.gz 2024-09-22T03:24:17.278 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.35756.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46323.log 2024-09-22T03:24:17.278 INFO:teuthology.orchestra.run.smithi153.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.35756.log.gz 2024-09-22T03:24:17.278 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.36506.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37532.log 2024-09-22T03:24:17.278 INFO:teuthology.orchestra.run.smithi153.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.36506.log.gz 2024-09-22T03:24:17.278 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.46323.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35804.log 2024-09-22T03:24:17.278 INFO:teuthology.orchestra.run.smithi153.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.46323.log.gz 2024-09-22T03:24:17.278 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49860.log 2024-09-22T03:24:17.278 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.37532.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.37532.log.gz 2024-09-22T03:24:17.278 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.35804.log: 50.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.35804.log.gz 2024-09-22T03:24:17.278 INFO:teuthology.orchestra.run.smithi153.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.35138.log 2024-09-22T03:24:17.279 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34151.log 2024-09-22T03:24:17.279 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.49860.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.49860.log.gz 2024-09-22T03:24:17.279 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.35138.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30994.log 2024-09-22T03:24:17.279 INFO:teuthology.orchestra.run.smithi153.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.35138.log.gz 2024-09-22T03:24:17.279 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49902.log 2024-09-22T03:24:17.279 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.34151.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.34151.log.gz 2024-09-22T03:24:17.279 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.30994.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.30994.log.gz 2024-09-22T03:24:17.279 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50111.log 2024-09-22T03:24:17.279 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35849.log 2024-09-22T03:24:17.279 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.49902.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.49902.log.gz 2024-09-22T03:24:17.279 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.50111.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40169.log 2024-09-22T03:24:17.279 INFO:teuthology.orchestra.run.smithi153.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.50111.log.gz 2024-09-22T03:24:17.279 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.35849.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28688.log 2024-09-22T03:24:17.279 INFO:teuthology.orchestra.run.smithi153.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.35849.log.gz 2024-09-22T03:24:17.279 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.40169.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36832.log 2024-09-22T03:24:17.279 INFO:teuthology.orchestra.run.smithi153.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.40169.log.gz 2024-09-22T03:24:17.280 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.28688.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.28688.log.gz 2024-09-22T03:24:17.280 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33762.log 2024-09-22T03:24:17.280 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.36832.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49944.log 2024-09-22T03:24:17.280 INFO:teuthology.orchestra.run.smithi153.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.36832.log.gz 2024-09-22T03:24:17.280 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.33762.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44955.log 2024-09-22T03:24:17.280 INFO:teuthology.orchestra.run.smithi153.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.33762.log.gz 2024-09-22T03:24:17.280 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32926.log 2024-09-22T03:24:17.280 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.49944.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.49944.log.gz 2024-09-22T03:24:17.280 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.44955.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32493.log 2024-09-22T03:24:17.280 INFO:teuthology.orchestra.run.smithi153.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.44955.log.gz 2024-09-22T03:24:17.280 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.32926.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38848.log 2024-09-22T03:24:17.280 INFO:teuthology.orchestra.run.smithi153.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.32926.log.gz 2024-09-22T03:24:17.280 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.32078.log 2024-09-22T03:24:17.280 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.32493.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.32493.log.gz 2024-09-22T03:24:17.280 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.38848.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.38848.log.gz 2024-09-22T03:24:17.281 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38766.log 2024-09-22T03:24:17.281 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.0.32078.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.35916.log 2024-09-22T03:24:17.281 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.38766.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.46957.log 2024-09-22T03:24:17.281 INFO:teuthology.orchestra.run.smithi153.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.38766.log.gz 2024-09-22T03:24:17.281 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.0.35916.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39681.log 2024-09-22T03:24:17.281 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.0.46957.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35221.log 2024-09-22T03:24:17.281 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.39681.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36683.log 2024-09-22T03:24:17.281 INFO:teuthology.orchestra.run.smithi153.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.39681.log.gz 2024-09-22T03:24:17.281 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.35221.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40517.log 2024-09-22T03:24:17.281 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.36683.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.35221.log.gz 2024-09-22T03:24:17.281 INFO:teuthology.orchestra.run.smithi153.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.36683.log.gz 2024-09-22T03:24:17.281 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49608.log 2024-09-22T03:24:17.281 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30043.log 2024-09-22T03:24:17.281 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.40517.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.40517.log.gz 2024-09-22T03:24:17.282 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.49608.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.49608.log.gz 2024-09-22T03:24:17.282 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29946.log 2024-09-22T03:24:17.282 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49692.log 2024-09-22T03:24:17.282 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.29946.log: /var/log/ceph/ceph-client.admin.30043.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.30043.log.gz 2024-09-22T03:24:17.282 INFO:teuthology.orchestra.run.smithi153.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29946.log.gz 2024-09-22T03:24:17.282 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34062.log 2024-09-22T03:24:17.282 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46749.log 2024-09-22T03:24:17.282 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.49692.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.49692.log.gz 2024-09-22T03:24:17.282 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.34062.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.34062.log.gz 2024-09-22T03:24:17.282 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40637.log 2024-09-22T03:24:17.282 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49776.log 2024-09-22T03:24:17.282 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.40637.log: /var/log/ceph/ceph-client.admin.46749.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.46749.log.gz 2024-09-22T03:24:17.283 INFO:teuthology.orchestra.run.smithi153.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.40637.log.gz 2024-09-22T03:24:17.283 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48558.log 2024-09-22T03:24:17.283 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49524.log 2024-09-22T03:24:17.283 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.49776.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.49776.log.gz 2024-09-22T03:24:17.283 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.48558.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.48558.log.gz 2024-09-22T03:24:17.283 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44390.log 2024-09-22T03:24:17.283 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47341.log 2024-09-22T03:24:17.283 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.44390.log: /var/log/ceph/ceph-client.admin.49524.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.49524.log.gz 2024-09-22T03:24:17.283 INFO:teuthology.orchestra.run.smithi153.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.44390.log.gz 2024-09-22T03:24:17.283 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39115.log 2024-09-22T03:24:17.283 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44470.log 2024-09-22T03:24:17.283 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.47341.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.47341.log.gz 2024-09-22T03:24:17.283 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.39115.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.39115.log.gz 2024-09-22T03:24:17.284 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30477.log 2024-09-22T03:24:17.284 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41626.log 2024-09-22T03:24:17.284 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.30477.log: /var/log/ceph/ceph-client.admin.44470.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.44470.log.gz 2024-09-22T03:24:17.284 INFO:teuthology.orchestra.run.smithi153.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30477.log.gz 2024-09-22T03:24:17.284 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-09-22T03:24:17.284 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31317.log 2024-09-22T03:24:17.284 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.41626.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.41626.log.gz 2024-09-22T03:24:17.284 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37579.log 2024-09-22T03:24:17.284 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.31317.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31317.log.gz 2024-09-22T03:24:17.284 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30131.log 2024-09-22T03:24:17.284 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.37579.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.37579.log.gz 2024-09-22T03:24:17.284 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31364.log 2024-09-22T03:24:17.284 INFO:teuthology.orchestra.run.smithi153.stderr: 88.1% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-09-22T03:24:17.284 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.30131.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38080.log 2024-09-22T03:24:17.284 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.31364.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30131.log.gz 2024-09-22T03:24:17.285 INFO:teuthology.orchestra.run.smithi153.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31364.log.gz 2024-09-22T03:24:17.285 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41931.log 2024-09-22T03:24:17.285 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35668.log 2024-09-22T03:24:17.285 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.38080.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.38080.log.gz 2024-09-22T03:24:17.285 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.41931.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.41931.log.gz 2024-09-22T03:24:17.285 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30259.log 2024-09-22T03:24:17.285 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44568.log 2024-09-22T03:24:17.285 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.30259.log: /var/log/ceph/ceph-client.admin.35668.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.35668.log.gz 2024-09-22T03:24:17.285 INFO:teuthology.orchestra.run.smithi153.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.30259.log.gz 2024-09-22T03:24:17.285 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37412.log 2024-09-22T03:24:17.285 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32779.log 2024-09-22T03:24:17.285 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.44568.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.44568.log.gz 2024-09-22T03:24:17.285 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.37412.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.37412.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33296.log 2024-09-22T03:24:17.285 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-22T03:24:17.285 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30045.log 2024-09-22T03:24:17.285 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.33296.log: /var/log/ceph/ceph-client.admin.32779.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.32779.log.gz 2024-09-22T03:24:17.286 INFO:teuthology.orchestra.run.smithi153.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.33296.log.gz 2024-09-22T03:24:17.286 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39899.log 2024-09-22T03:24:17.286 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35390.log 2024-09-22T03:24:17.286 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.30045.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.30045.log.gz 2024-09-22T03:24:17.286 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.39899.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.39899.log.gz 2024-09-22T03:24:17.286 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44915.log 2024-09-22T03:24:17.286 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.35390.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.35390.log.gz 2024-09-22T03:24:17.286 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41158.log 2024-09-22T03:24:17.286 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.30331.log 2024-09-22T03:24:17.286 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.44915.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.44915.log.gz 2024-09-22T03:24:17.286 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.41158.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.41158.log.gz 2024-09-22T03:24:17.286 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47014.log 2024-09-22T03:24:17.286 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41031.log 2024-09-22T03:24:17.286 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.47014.log: /var/log/ceph/ceph-client.0.30331.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.47014.log.gz 2024-09-22T03:24:17.287 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31040.log 2024-09-22T03:24:17.287 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.41031.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.41031.log.gz 2024-09-22T03:24:17.287 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33637.log 2024-09-22T03:24:17.287 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.31040.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31040.log.gz 2024-09-22T03:24:17.287 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37492.log 2024-09-22T03:24:17.287 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.33637.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.33637.log.gz 2024-09-22T03:24:17.287 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29590.log 2024-09-22T03:24:17.287 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.37492.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.37492.log.gz 2024-09-22T03:24:17.287 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49290.log 2024-09-22T03:24:17.287 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.29590.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29590.log.gz 2024-09-22T03:24:17.287 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41581.log 2024-09-22T03:24:17.287 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.49290.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.49290.log.gz 2024-09-22T03:24:17.287 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37452.log 2024-09-22T03:24:17.287 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.41581.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41581.log.gz 2024-09-22T03:24:17.287 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34231.log 2024-09-22T03:24:17.288 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.37452.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.37452.log.gz 2024-09-22T03:24:17.288 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33443.log 2024-09-22T03:24:17.288 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.34231.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.34231.log.gz 2024-09-22T03:24:17.288 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32135.log 2024-09-22T03:24:17.288 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.33443.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.33443.log.gz 2024-09-22T03:24:17.288 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41472.log 2024-09-22T03:24:17.288 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.32135.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.32135.log.gz 2024-09-22T03:24:17.288 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49291.log 2024-09-22T03:24:17.288 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.41472.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41472.log.gz 2024-09-22T03:24:17.288 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35973.log 2024-09-22T03:24:17.288 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.49291.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.49291.log.gz 2024-09-22T03:24:17.288 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45795.log 2024-09-22T03:24:17.288 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.35973.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.35973.log.gz 2024-09-22T03:24:17.288 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39859.log 2024-09-22T03:24:17.288 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.45795.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.45795.log.gz 2024-09-22T03:24:17.288 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39641.log 2024-09-22T03:24:17.288 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.39859.log: 49.2% -- replaced with /var/log/ceph/ceph-client.admin.39859.log.gz 2024-09-22T03:24:17.289 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49734.log 2024-09-22T03:24:17.289 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.39641.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.39641.log.gz 2024-09-22T03:24:17.289 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29128.log 2024-09-22T03:24:17.289 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.49734.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.49734.log.gz 2024-09-22T03:24:17.289 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32355.log 2024-09-22T03:24:17.289 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.29128.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.29128.log.gz 2024-09-22T03:24:17.289 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43722.log 2024-09-22T03:24:17.289 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.32355.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.32355.log.gz 2024-09-22T03:24:17.289 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48642.log 2024-09-22T03:24:17.289 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.43722.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.43722.log.gz 2024-09-22T03:24:17.289 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34411.log 2024-09-22T03:24:17.289 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.48642.log: 49.2% -- replaced with /var/log/ceph/ceph-client.admin.48642.log.gz 2024-09-22T03:24:17.289 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40735.log 2024-09-22T03:24:17.289 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.34411.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.34411.log.gz 2024-09-22T03:24:17.289 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30217.log 2024-09-22T03:24:17.289 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.40735.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.40735.log.gz 2024-09-22T03:24:17.290 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41285.log 2024-09-22T03:24:17.290 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.30217.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.30217.log.gz 2024-09-22T03:24:17.290 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35264.log 2024-09-22T03:24:17.290 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.41285.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41285.log.gz 2024-09-22T03:24:17.290 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46616.log 2024-09-22T03:24:17.290 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.35264.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.35264.log.gz 2024-09-22T03:24:17.290 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42997.log 2024-09-22T03:24:17.290 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.46616.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.46616.log.gz 2024-09-22T03:24:17.290 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33884.log 2024-09-22T03:24:17.290 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.42997.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.42997.log.gz 2024-09-22T03:24:17.290 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41531.log 2024-09-22T03:24:17.290 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.33884.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.33884.log.gz 2024-09-22T03:24:17.290 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39819.log 2024-09-22T03:24:17.290 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.41531.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41531.log.gz 2024-09-22T03:24:17.291 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33345.log 2024-09-22T03:24:17.291 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.39819.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.39819.log.gz 2024-09-22T03:24:17.291 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44875.log 2024-09-22T03:24:17.291 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.33345.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.33345.log.gz 2024-09-22T03:24:17.291 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48852.log 2024-09-22T03:24:17.291 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.44875.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.44875.log.gz 2024-09-22T03:24:17.291 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49440.log 2024-09-22T03:24:17.291 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.48852.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.48852.log.gz 2024-09-22T03:24:17.291 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32698.log 2024-09-22T03:24:17.291 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.49440.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.49440.log.gz 2024-09-22T03:24:17.291 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37114.log 2024-09-22T03:24:17.291 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.32698.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.32698.log.gz 2024-09-22T03:24:17.291 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43179.log 2024-09-22T03:24:17.291 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.37114.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.37114.log.gz 2024-09-22T03:24:17.291 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39293.log 2024-09-22T03:24:17.292 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.43179.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.43179.log.gz 2024-09-22T03:24:17.292 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37067.log 2024-09-22T03:24:17.292 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.39293.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.39293.log.gz 2024-09-22T03:24:17.292 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36136.log 2024-09-22T03:24:17.292 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.37067.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.37067.log.gz 2024-09-22T03:24:17.292 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36274.log 2024-09-22T03:24:17.292 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.36136.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.36136.log.gz 2024-09-22T03:24:17.292 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34460.log 2024-09-22T03:24:17.292 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.36274.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.36274.log.gz 2024-09-22T03:24:17.292 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42352.log 2024-09-22T03:24:17.292 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.34460.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.34460.log.gz 2024-09-22T03:24:17.292 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29440.log 2024-09-22T03:24:17.292 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.42352.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.42352.log.gz 2024-09-22T03:24:17.292 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48516.log 2024-09-22T03:24:17.292 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.29440.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29440.log.gz 2024-09-22T03:24:17.292 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48298.log 2024-09-22T03:24:17.293 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.48516.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.48516.log.gz 2024-09-22T03:24:17.293 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.37783.log 2024-09-22T03:24:17.293 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.48298.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.48298.log.gz 2024-09-22T03:24:17.293 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40302.log 2024-09-22T03:24:17.295 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.0.37783.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32395.log 2024-09-22T03:24:17.295 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.40302.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.40302.log.gz 2024-09-22T03:24:17.295 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47527.log 2024-09-22T03:24:17.295 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.32395.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.32395.log.gz 2024-09-22T03:24:17.295 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48952.log 2024-09-22T03:24:17.295 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.47527.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.47527.log.gz 2024-09-22T03:24:17.298 INFO:teuthology.orchestra.run.smithi196.stderr:/var/log/ceph/ceph-mgr.x.log: /var/log/ceph/ceph-osd.3.log: 91.7% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-09-22T03:24:17.304 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40029.log 2024-09-22T03:24:17.304 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.48952.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.48952.log.gz 2024-09-22T03:24:17.321 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34678.log 2024-09-22T03:24:17.321 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.40029.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.40029.log.gz 2024-09-22T03:24:17.330 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48953.log 2024-09-22T03:24:17.330 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.34678.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.34678.log.gz 2024-09-22T03:24:17.349 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48954.log 2024-09-22T03:24:17.349 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.48953.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.48953.log.gz 2024-09-22T03:24:17.364 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30301.log 2024-09-22T03:24:17.364 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.48954.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.48954.log.gz 2024-09-22T03:24:17.373 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39427.log 2024-09-22T03:24:17.373 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.30301.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30301.log.gz 2024-09-22T03:24:17.382 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45035.log 2024-09-22T03:24:17.382 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.39427.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.39427.log.gz 2024-09-22T03:24:17.391 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32828.log 2024-09-22T03:24:17.391 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.45035.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.45035.log.gz 2024-09-22T03:24:17.403 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31178.log 2024-09-22T03:24:17.403 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.32828.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.32828.log.gz 2024-09-22T03:24:17.417 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43275.log 2024-09-22T03:24:17.417 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.31178.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.31178.log.gz 2024-09-22T03:24:17.437 INFO:teuthology.orchestra.run.smithi153.stderr: 94.2% -- replaced with /var/log/ceph/ceph-client.0.35916.log.gz 2024-09-22T03:24:17.438 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47716.log 2024-09-22T03:24:17.439 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.43275.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.43275.log.gz 2024-09-22T03:24:17.439 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35344.log 2024-09-22T03:24:17.439 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.47716.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.47716.log.gz 2024-09-22T03:24:17.441 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44212.log 2024-09-22T03:24:17.441 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.35344.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.35344.log.gz 2024-09-22T03:24:17.441 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43673.log 2024-09-22T03:24:17.442 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.44212.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.44212.log.gz 2024-09-22T03:24:17.443 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46518.log 2024-09-22T03:24:17.443 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.43673.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.43673.log.gz 2024-09-22T03:24:17.444 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42647.log 2024-09-22T03:24:17.444 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.46518.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.46518.log.gz 2024-09-22T03:24:17.444 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46658.log 2024-09-22T03:24:17.446 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.42647.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.42647.log.gz 2024-09-22T03:24:17.446 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38633.log 2024-09-22T03:24:17.446 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.46658.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.46658.log.gz 2024-09-22T03:24:17.448 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36366.log 2024-09-22T03:24:17.448 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.38633.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.38633.log.gz 2024-09-22T03:24:17.448 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49119.log 2024-09-22T03:24:17.450 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.36366.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.36366.log.gz 2024-09-22T03:24:17.450 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34369.log 2024-09-22T03:24:17.450 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.49119.log: 50.8% -- replaced with /var/log/ceph/ceph-client.admin.49119.log.gz 2024-09-22T03:24:17.450 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39561.log 2024-09-22T03:24:17.451 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.34369.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.34369.log.gz 2024-09-22T03:24:17.451 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46163.log 2024-09-22T03:24:17.452 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.39561.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.39561.log.gz 2024-09-22T03:24:17.453 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42688.log 2024-09-22T03:24:17.453 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.46163.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.46163.log.gz 2024-09-22T03:24:17.453 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38724.log 2024-09-22T03:24:17.456 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.42688.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.42688.log.gz 2024-09-22T03:24:17.456 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45617.log 2024-09-22T03:24:17.456 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.38724.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.38724.log.gz 2024-09-22T03:24:17.456 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43807.log 2024-09-22T03:24:17.456 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.45617.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.45617.log.gz 2024-09-22T03:24:17.458 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33394.log 2024-09-22T03:24:17.458 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.43807.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.43807.log.gz 2024-09-22T03:24:17.458 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49482.log 2024-09-22T03:24:17.459 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.33394.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.33394.log.gz 2024-09-22T03:24:17.459 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44708.log 2024-09-22T03:24:17.459 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.49482.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.49482.log.gz 2024-09-22T03:24:17.461 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28532.log 2024-09-22T03:24:17.461 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.44708.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.44708.log.gz 2024-09-22T03:24:17.461 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28715.log 2024-09-22T03:24:17.462 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.28532.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.28532.log.gz 2024-09-22T03:24:17.462 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42286.log 2024-09-22T03:24:17.463 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.28715.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.28715.log.gz 2024-09-22T03:24:17.464 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29399.log 2024-09-22T03:24:17.464 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.42286.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.42286.log.gz 2024-09-22T03:24:17.464 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38888.log 2024-09-22T03:24:17.465 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.29399.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29399.log.gz 2024-09-22T03:24:17.466 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35715.log 2024-09-22T03:24:17.466 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.38888.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.38888.log.gz 2024-09-22T03:24:17.467 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49986.log 2024-09-22T03:24:17.467 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.35715.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.35715.log.gz 2024-09-22T03:24:17.468 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31132.log 2024-09-22T03:24:17.468 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.49986.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.49986.log.gz 2024-09-22T03:24:17.469 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38808.log 2024-09-22T03:24:17.469 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.31132.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.31132.log.gz 2024-09-22T03:24:17.470 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37841.log 2024-09-22T03:24:17.470 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.38808.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.38808.log.gz 2024-09-22T03:24:17.471 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35178.log 2024-09-22T03:24:17.471 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.37841.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.37841.log.gz 2024-09-22T03:24:17.473 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43226.log 2024-09-22T03:24:17.473 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.35178.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.35178.log.gz 2024-09-22T03:24:17.473 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42842.log 2024-09-22T03:24:17.474 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.43226.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.43226.log.gz 2024-09-22T03:24:17.474 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34718.log 2024-09-22T03:24:17.475 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.42842.log: 49.2% -- replaced with /var/log/ceph/ceph-client.admin.42842.log.gz 2024-09-22T03:24:17.476 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43631.log 2024-09-22T03:24:17.476 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.34718.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.34718.log.gz 2024-09-22T03:24:17.477 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-09-22T03:24:17.477 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.43631.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.43631.log.gz 2024-09-22T03:24:17.477 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29439.log 2024-09-22T03:24:17.487 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-mon.a.log: 94.1% -- replaced with /var/log/ceph/ceph-client.0.46957.log.gz 2024-09-22T03:24:17.488 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39017.log 2024-09-22T03:24:17.489 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.29439.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.29439.log.gz 2024-09-22T03:24:17.489 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46421.log 2024-09-22T03:24:17.489 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.39017.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.39017.log.gz 2024-09-22T03:24:17.490 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40127.log 2024-09-22T03:24:17.490 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.46421.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.46421.log.gz 2024-09-22T03:24:17.491 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49122.log 2024-09-22T03:24:17.492 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.40127.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.40127.log.gz 2024-09-22T03:24:17.493 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49062.log 2024-09-22T03:24:17.493 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.49122.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.49122.log.gz 2024-09-22T03:24:17.494 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36412.log 2024-09-22T03:24:17.494 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.49062.log: 50.8% -- replaced with /var/log/ceph/ceph-client.admin.49062.log.gz 2024-09-22T03:24:17.495 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29086.log 2024-09-22T03:24:17.495 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.36412.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.36412.log.gz 2024-09-22T03:24:17.496 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30517.log 2024-09-22T03:24:17.496 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.29086.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29086.log.gz 2024-09-22T03:24:17.497 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42955.log 2024-09-22T03:24:17.497 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.30517.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.30517.log.gz 2024-09-22T03:24:17.499 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37234.log 2024-09-22T03:24:17.499 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.42955.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.42955.log.gz 2024-09-22T03:24:17.499 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32235.log 2024-09-22T03:24:17.500 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.37234.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.37234.log.gz 2024-09-22T03:24:17.502 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41980.log 2024-09-22T03:24:17.502 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.32235.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.32235.log.gz 2024-09-22T03:24:17.502 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32877.log 2024-09-22T03:24:17.502 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.41980.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41980.log.gz 2024-09-22T03:24:17.504 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41238.log 2024-09-22T03:24:17.504 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.32877.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.32877.log.gz 2024-09-22T03:24:17.504 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31270.log 2024-09-22T03:24:17.505 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.41238.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.41238.log.gz 2024-09-22T03:24:17.505 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34271.log 2024-09-22T03:24:17.505 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.31270.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31270.log.gz 2024-09-22T03:24:17.507 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37888.log 2024-09-22T03:24:17.507 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.34271.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.34271.log.gz 2024-09-22T03:24:17.508 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43493.log 2024-09-22T03:24:17.508 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.37888.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.37888.log.gz 2024-09-22T03:24:17.509 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34320.log 2024-09-22T03:24:17.509 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.43493.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.43493.log.gz 2024-09-22T03:24:17.509 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39477.log 2024-09-22T03:24:17.510 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.34320.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.34320.log.gz 2024-09-22T03:24:17.511 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46022.log 2024-09-22T03:24:17.512 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.39477.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.39477.log.gz 2024-09-22T03:24:17.512 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48600.log 2024-09-22T03:24:17.514 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.46022.log: 94.1% 50.0% -- replaced with /var/log/ceph/ceph-client.admin.46022.log.gz 2024-09-22T03:24:17.514 INFO:teuthology.orchestra.run.smithi153.stderr: -- replaced with /var/log/ceph/ceph-client.0.30331.log.gz 2024-09-22T03:24:17.514 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29027.log 2024-09-22T03:24:17.514 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.48600.log: /var/log/ceph/ceph-client.admin.29842.log 2024-09-22T03:24:17.514 INFO:teuthology.orchestra.run.smithi153.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.48600.log.gz 2024-09-22T03:24:17.514 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.29027.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45497.log 2024-09-22T03:24:17.514 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.29842.log: 66.7% -- replaced with /var/log/ceph/ceph-client.admin.29027.log.gz 2024-09-22T03:24:17.515 INFO:teuthology.orchestra.run.smithi153.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.29842.log.gz 2024-09-22T03:24:17.515 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47975.log 2024-09-22T03:24:17.530 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36320.log 2024-09-22T03:24:17.530 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.45497.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.45497.log.gz 2024-09-22T03:24:17.530 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.47975.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.47975.log.gz 2024-09-22T03:24:17.530 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45457.log 2024-09-22T03:24:17.530 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43324.log 2024-09-22T03:24:17.530 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.45457.log: /var/log/ceph/ceph-client.admin.36320.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.36320.log.gz 2024-09-22T03:24:17.530 INFO:teuthology.orchestra.run.smithi153.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.45457.log.gz 2024-09-22T03:24:17.530 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28944.log 2024-09-22T03:24:17.530 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5/var/log/ceph/ceph-client.admin.43324.log: --verbose -- /var/log/ceph/ceph-client.admin.47576.log 2024-09-22T03:24:17.530 INFO:teuthology.orchestra.run.smithi153.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.43324.log.gz 2024-09-22T03:24:17.530 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.28944.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39919.log 2024-09-22T03:24:17.530 INFO:teuthology.orchestra.run.smithi153.stderr: 49.6%/var/log/ceph/ceph-client.admin.47576.log: -- replaced with /var/log/ceph/ceph-client.admin.28944.log.gz 2024-09-22T03:24:17.531 INFO:teuthology.orchestra.run.smithi153.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.47576.log.gz 2024-09-22T03:24:17.531 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30908.log 2024-09-22T03:24:17.531 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33490.log 2024-09-22T03:24:17.531 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.39919.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.39919.log.gz 2024-09-22T03:24:17.531 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.30908.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30908.log.gz 2024-09-22T03:24:17.531 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41667.log 2024-09-22T03:24:17.531 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.33490.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48249.log 2024-09-22T03:24:17.531 INFO:teuthology.orchestra.run.smithi153.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.33490.log.gz 2024-09-22T03:24:17.531 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.41667.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43858.log 2024-09-22T03:24:17.531 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.48249.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.48249.log.gz 2024-09-22T03:24:17.531 INFO:teuthology.orchestra.run.smithi153.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41667.log.gz 2024-09-22T03:24:17.531 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34767.log 2024-09-22T03:24:17.531 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.43858.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36969.log 2024-09-22T03:24:17.532 INFO:teuthology.orchestra.run.smithi153.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.43858.log.gz 2024-09-22T03:24:17.532 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.34767.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40078.log 2024-09-22T03:24:17.532 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.36969.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.34767.log.gz 2024-09-22T03:24:17.532 INFO:teuthology.orchestra.run.smithi153.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.36969.log.gz 2024-09-22T03:24:17.532 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38499.log 2024-09-22T03:24:17.532 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.40078.log: -- /var/log/ceph/ceph-client.admin.29758.log 2024-09-22T03:24:17.532 INFO:teuthology.orchestra.run.smithi153.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.40078.log.gz 2024-09-22T03:24:17.532 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.38499.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39721.log 2024-09-22T03:24:17.532 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.29758.log: 49.6% 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29758.log.gz 2024-09-22T03:24:17.532 INFO:teuthology.orchestra.run.smithi153.stderr: -- replaced with /var/log/ceph/ceph-client.admin.38499.log.gz 2024-09-22T03:24:17.532 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43344.log 2024-09-22T03:24:17.532 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.39721.log: 50.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43900.log 2024-09-22T03:24:17.532 INFO:teuthology.orchestra.run.smithi153.stderr: -- replaced with /var/log/ceph/ceph-client.admin.39721.log.gz 2024-09-22T03:24:17.532 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40944.log 2024-09-22T03:24:17.533 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.43344.log: /var/log/ceph/ceph-client.admin.43900.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.43900.log.gz 2024-09-22T03:24:17.533 INFO:teuthology.orchestra.run.smithi153.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.43344.log.gz 2024-09-22T03:24:17.533 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31964.log 2024-09-22T03:24:17.533 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.40944.log: 50.4%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46792.log 2024-09-22T03:24:17.533 INFO:teuthology.orchestra.run.smithi153.stderr: -- replaced with /var/log/ceph/ceph-client.admin.40944.log.gz 2024-09-22T03:24:17.533 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41800.log 2024-09-22T03:24:17.533 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.31964.log: /var/log/ceph/ceph-client.admin.46792.log: 49.6% 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31964.log.gz 2024-09-22T03:24:17.533 INFO:teuthology.orchestra.run.smithi153.stderr: -- replaced with /var/log/ceph/ceph-client.admin.46792.log.gz 2024-09-22T03:24:17.533 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34107.log 2024-09-22T03:24:17.533 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44648.log 2024-09-22T03:24:17.533 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.41800.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41800.log.gz 2024-09-22T03:24:17.533 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.34107.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.34107.log.gz 2024-09-22T03:24:17.533 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33842.log 2024-09-22T03:24:17.533 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42203.log 2024-09-22T03:24:17.534 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.44648.log: /var/log/ceph/ceph-client.admin.33842.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.44648.log.gz 2024-09-22T03:24:17.534 INFO:teuthology.orchestra.run.smithi153.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.33842.log.gz 2024-09-22T03:24:17.534 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.33956.log 2024-09-22T03:24:17.534 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42118.log 2024-09-22T03:24:17.535 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.42203.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.42203.log.gz 2024-09-22T03:24:17.535 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.0.33956.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48431.log 2024-09-22T03:24:17.535 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.42118.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.42118.log.gz 2024-09-22T03:24:17.536 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-09-22T03:24:17.536 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.48431.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.48431.log.gz 2024-09-22T03:24:17.536 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48200.log 2024-09-22T03:24:17.536 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph.log: 86.9% -- replaced with /var/log/ceph/ceph.log.gz 2024-09-22T03:24:17.536 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44310.log 2024-09-22T03:24:17.537 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.48200.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.48200.log.gz 2024-09-22T03:24:17.537 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35574.log 2024-09-22T03:24:17.537 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.44310.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.44310.log.gz 2024-09-22T03:24:17.538 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31224.log 2024-09-22T03:24:17.538 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.35574.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.35574.log.gz 2024-09-22T03:24:17.539 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36459.log 2024-09-22T03:24:17.539 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.31224.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.31224.log.gz 2024-09-22T03:24:17.539 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33200.log 2024-09-22T03:24:17.540 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.36459.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.36459.log.gz 2024-09-22T03:24:17.540 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38129.log 2024-09-22T03:24:17.540 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.33200.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.33200.log.gz 2024-09-22T03:24:17.541 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46927.log 2024-09-22T03:24:17.541 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.38129.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.38129.log.gz 2024-09-22T03:24:17.542 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38031.log 2024-09-22T03:24:17.542 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.46927.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.46927.log.gz 2024-09-22T03:24:17.543 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44995.log 2024-09-22T03:24:17.543 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.38031.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.38031.log.gz 2024-09-22T03:24:17.544 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.41696.log 2024-09-22T03:24:17.544 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.44995.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.44995.log.gz 2024-09-22T03:24:17.544 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44261.log 2024-09-22T03:24:17.553 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.0.41696.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40431.log 2024-09-22T03:24:17.554 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.44261.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.44261.log.gz 2024-09-22T03:24:17.567 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29000.log 2024-09-22T03:24:17.568 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.40431.log: 49.2% -- replaced with /var/log/ceph/ceph-client.admin.40431.log.gz 2024-09-22T03:24:17.568 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47665.log 2024-09-22T03:24:17.569 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.29000.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29000.log.gz 2024-09-22T03:24:17.576 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35528.log 2024-09-22T03:24:17.577 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.47665.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.47665.log.gz 2024-09-22T03:24:17.594 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31710.log 2024-09-22T03:24:17.594 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.35528.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.35528.log.gz 2024-09-22T03:24:17.603 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30868.log 2024-09-22T03:24:17.603 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.31710.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31710.log.gz 2024-09-22T03:24:17.612 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45715.log 2024-09-22T03:24:17.612 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.30868.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.30868.log.gz 2024-09-22T03:24:17.623 INFO:teuthology.orchestra.run.smithi153.stderr: 94.1% -- replaced with /var/log/ceph/ceph-client.0.32078.log.gz 2024-09-22T03:24:17.624 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47063.log 2024-09-22T03:24:17.624 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.45715.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.45715.log.gz 2024-09-22T03:24:17.624 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44777.log 2024-09-22T03:24:17.625 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.47063.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.47063.log.gz 2024-09-22T03:24:17.628 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33539.log 2024-09-22T03:24:17.628 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.44777.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.44777.log.gz 2024-09-22T03:24:17.629 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41886.log 2024-09-22T03:24:17.629 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.33539.log: 49.2% -- replaced with /var/log/ceph/ceph-client.admin.33539.log.gz 2024-09-22T03:24:17.629 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49818.log 2024-09-22T03:24:17.629 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.41886.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41886.log.gz 2024-09-22T03:24:17.629 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44688.log 2024-09-22T03:24:17.629 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.49818.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.49818.log.gz 2024-09-22T03:24:17.629 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38968.log 2024-09-22T03:24:17.631 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.44688.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.44688.log.gz 2024-09-22T03:24:17.631 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42536.log 2024-09-22T03:24:17.631 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.38968.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.38968.log.gz 2024-09-22T03:24:17.631 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34610.log 2024-09-22T03:24:17.631 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.42536.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.42536.log.gz 2024-09-22T03:24:17.633 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31557.log 2024-09-22T03:24:17.633 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.34610.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.34610.log.gz 2024-09-22T03:24:17.633 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42605.log 2024-09-22T03:24:17.634 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.31557.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.31557.log.gz 2024-09-22T03:24:17.635 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42913.log 2024-09-22T03:24:17.635 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.42605.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.42605.log.gz 2024-09-22T03:24:17.636 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46203.log 2024-09-22T03:24:17.636 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.42913.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.42913.log.gz 2024-09-22T03:24:17.636 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41452.log 2024-09-22T03:24:17.639 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.46203.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.46203.log.gz 2024-09-22T03:24:17.639 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44826.log 2024-09-22T03:24:17.639 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.41452.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41452.log.gz 2024-09-22T03:24:17.639 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45577.log 2024-09-22T03:24:17.639 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.44826.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.44826.log.gz 2024-09-22T03:24:17.641 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32315.log 2024-09-22T03:24:17.641 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.45577.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.45577.log.gz 2024-09-22T03:24:17.641 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30695.log 2024-09-22T03:24:17.642 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.32315.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.32315.log.gz 2024-09-22T03:24:17.642 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37713.log 2024-09-22T03:24:17.642 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.30695.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30695.log.gz 2024-09-22T03:24:17.644 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34549.log 2024-09-22T03:24:17.644 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.37713.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.37713.log.gz 2024-09-22T03:24:17.645 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29632.log 2024-09-22T03:24:17.645 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.34549.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.34549.log.gz 2024-09-22T03:24:17.646 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39601.log 2024-09-22T03:24:17.646 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.29632.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29632.log.gz 2024-09-22T03:24:17.646 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41392.log 2024-09-22T03:24:17.647 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.39601.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.39601.log.gz 2024-09-22T03:24:17.647 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46372.log 2024-09-22T03:24:17.648 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.41392.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41392.log.gz 2024-09-22T03:24:17.649 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30557.log 2024-09-22T03:24:17.650 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.46372.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.46372.log.gz 2024-09-22T03:24:17.650 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49398.log 2024-09-22T03:24:17.650 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.30557.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30557.log.gz 2024-09-22T03:24:17.652 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38220.log 2024-09-22T03:24:17.652 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.49398.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.49398.log.gz 2024-09-22T03:24:17.652 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45133.log 2024-09-22T03:24:17.653 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.38220.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.38220.log.gz 2024-09-22T03:24:17.653 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44430.log 2024-09-22T03:24:17.655 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.45133.log: 49.2% -- replaced with /var/log/ceph/ceph-client.admin.45133.log.gz 2024-09-22T03:24:17.655 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44608.log 2024-09-22T03:24:17.655 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.44430.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.44430.log.gz 2024-09-22T03:24:17.655 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46707.log 2024-09-22T03:24:17.657 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.44608.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.44608.log.gz 2024-09-22T03:24:17.657 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40864.log 2024-09-22T03:24:17.657 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.46707.log: 48.0% -- replaced with /var/log/ceph/ceph-client.admin.46707.log.gz 2024-09-22T03:24:17.657 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.27377.log 2024-09-22T03:24:17.659 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.40864.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.40864.log.gz 2024-09-22T03:24:17.659 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49566.log 2024-09-22T03:24:17.659 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph.tmp-client.admin.27377.log: 68.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.27377.log.gz 2024-09-22T03:24:17.661 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31779.log 2024-09-22T03:24:17.661 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.49566.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.49566.log.gz 2024-09-22T03:24:17.661 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29272.log 2024-09-22T03:24:17.662 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.31779.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.31779.log.gz 2024-09-22T03:24:17.662 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38171.log 2024-09-22T03:24:17.662 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.29272.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.29272.log.gz 2024-09-22T03:24:17.664 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40477.log 2024-09-22T03:24:17.664 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.38171.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.38171.log.gz 2024-09-22T03:24:17.664 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44519.log 2024-09-22T03:24:17.665 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.40477.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.40477.log.gz 2024-09-22T03:24:17.665 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44350.log 2024-09-22T03:24:17.666 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.44519.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.44519.log.gz 2024-09-22T03:24:17.667 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37194.log 2024-09-22T03:24:17.667 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.44350.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.44350.log.gz 2024-09-22T03:24:17.667 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40782.log 2024-09-22T03:24:17.669 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.37194.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.37194.log.gz 2024-09-22T03:24:17.669 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29400.log 2024-09-22T03:24:17.669 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.40782.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.40782.log.gz 2024-09-22T03:24:17.670 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38409.log 2024-09-22T03:24:17.670 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.29400.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.29400.log.gz 2024-09-22T03:24:17.671 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30388.log 2024-09-22T03:24:17.671 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.38409.log: 49.2% -- replaced with /var/log/ceph/ceph-client.admin.38409.log.gz 2024-09-22T03:24:17.672 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46065.log 2024-09-22T03:24:17.673 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.30388.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.30388.log.gz 2024-09-22T03:24:17.673 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42738.log 2024-09-22T03:24:17.674 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.46065.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.46065.log.gz 2024-09-22T03:24:17.674 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37278.log 2024-09-22T03:24:17.675 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.42738.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.42738.log.gz 2024-09-22T03:24:17.675 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36053.log 2024-09-22T03:24:17.675 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.37278.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.37278.log.gz 2024-09-22T03:24:17.677 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28453.log 2024-09-22T03:24:17.677 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.36053.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.36053.log.gz 2024-09-22T03:24:17.677 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38360.log 2024-09-22T03:24:17.678 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.28453.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.28453.log.gz 2024-09-22T03:24:17.678 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33722.log 2024-09-22T03:24:17.680 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.38360.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.38360.log.gz 2024-09-22T03:24:17.680 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36920.log 2024-09-22T03:24:17.680 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.33722.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.33722.log.gz 2024-09-22T03:24:17.681 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43582.log 2024-09-22T03:24:17.682 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.36920.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.36920.log.gz 2024-09-22T03:24:17.682 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40824.log 2024-09-22T03:24:17.684 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.43582.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.43582.log.gz 2024-09-22T03:24:17.684 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33151.log 2024-09-22T03:24:17.684 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.40824.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.40824.log.gz 2024-09-22T03:24:17.684 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40260.log 2024-09-22T03:24:17.685 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.33151.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.33151.log.gz 2024-09-22T03:24:17.685 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42780.log 2024-09-22T03:24:17.685 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.40260.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.40260.log.gz 2024-09-22T03:24:17.687 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35482.log 2024-09-22T03:24:17.687 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.42780.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.42780.log.gz 2024-09-22T03:24:17.687 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43533.log 2024-09-22T03:24:17.689 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.35482.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.35482.log.gz 2024-09-22T03:24:17.689 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41372.log 2024-09-22T03:24:17.689 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.43533.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.43533.log.gz 2024-09-22T03:24:17.690 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38458.log 2024-09-22T03:24:17.690 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.41372.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.41372.log.gz 2024-09-22T03:24:17.690 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46243.log 2024-09-22T03:24:17.692 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.38458.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.38458.log.gz 2024-09-22T03:24:17.692 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43764.log 2024-09-22T03:24:17.692 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.46243.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.46243.log.gz 2024-09-22T03:24:17.693 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50071.log 2024-09-22T03:24:17.694 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.43764.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.43764.log.gz 2024-09-22T03:24:17.694 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29800.log 2024-09-22T03:24:17.695 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.50071.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.50071.log.gz 2024-09-22T03:24:17.695 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38673.log 2024-09-22T03:24:17.695 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.29800.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.29800.log.gz 2024-09-22T03:24:17.696 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36597.log 2024-09-22T03:24:17.696 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.38673.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.38673.log.gz 2024-09-22T03:24:17.698 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30782.log 2024-09-22T03:24:17.698 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.36597.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.36597.log.gz 2024-09-22T03:24:17.698 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37018.log 2024-09-22T03:24:17.699 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.30782.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30782.log.gz 2024-09-22T03:24:17.700 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40686.log 2024-09-22T03:24:17.700 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.37018.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.37018.log.gz 2024-09-22T03:24:17.701 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34569.log 2024-09-22T03:24:17.701 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.40686.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.40686.log.gz 2024-09-22T03:24:17.702 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32275.log 2024-09-22T03:24:17.702 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.34569.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.34569.log.gz 2024-09-22T03:24:17.702 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40597.log 2024-09-22T03:24:17.704 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.32275.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.32275.log.gz 2024-09-22T03:24:17.704 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47891.log 2024-09-22T03:24:17.704 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.40597.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.40597.log.gz 2024-09-22T03:24:17.706 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39988.log 2024-09-22T03:24:17.706 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.47891.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.47891.log.gz 2024-09-22T03:24:17.706 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30597.log 2024-09-22T03:24:17.708 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.39988.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.39988.log.gz 2024-09-22T03:24:17.708 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31870.log 2024-09-22T03:24:17.708 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.30597.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.30597.log.gz 2024-09-22T03:24:17.708 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48726.log 2024-09-22T03:24:17.709 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.31870.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31870.log.gz 2024-09-22T03:24:17.710 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37325.log 2024-09-22T03:24:17.710 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.48726.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.48726.log.gz 2024-09-22T03:24:17.712 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31471.log 2024-09-22T03:24:17.712 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.37325.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.37325.log.gz 2024-09-22T03:24:17.712 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45666.log 2024-09-22T03:24:17.712 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.31471.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.31471.log.gz 2024-09-22T03:24:17.713 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33679.log 2024-09-22T03:24:17.713 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.45666.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.45666.log.gz 2024-09-22T03:24:17.714 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45537.log 2024-09-22T03:24:17.715 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.33679.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.33679.log.gz 2024-09-22T03:24:17.715 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30742.log 2024-09-22T03:24:17.715 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.45537.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.45537.log.gz 2024-09-22T03:24:17.716 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41332.log 2024-09-22T03:24:17.716 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.30742.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.30742.log.gz 2024-09-22T03:24:17.717 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48768.log 2024-09-22T03:24:17.717 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.41332.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.41332.log.gz 2024-09-22T03:24:17.719 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46114.log 2024-09-22T03:24:17.719 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.48768.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.48768.log.gz 2024-09-22T03:24:17.719 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42822.log 2024-09-22T03:24:17.720 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.46114.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.46114.log.gz 2024-09-22T03:24:17.721 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39386.log 2024-09-22T03:24:17.721 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.42822.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.42822.log.gz 2024-09-22T03:24:17.722 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50028.log 2024-09-22T03:24:17.722 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.39386.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.39386.log.gz 2024-09-22T03:24:17.722 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30174.log 2024-09-22T03:24:17.724 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.50028.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.50028.log.gz 2024-09-22T03:24:17.724 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41844.log 2024-09-22T03:24:17.724 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.30174.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.30174.log.gz 2024-09-22T03:24:17.725 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32444.log 2024-09-22T03:24:17.725 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.41844.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41844.log.gz 2024-09-22T03:24:17.727 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39066.log 2024-09-22T03:24:17.727 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.32444.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.32444.log.gz 2024-09-22T03:24:17.727 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42027.log 2024-09-22T03:24:17.728 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.39066.log: 49.2% -- replaced with /var/log/ceph/ceph-client.admin.39066.log.gz 2024-09-22T03:24:17.728 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30646.log 2024-09-22T03:24:17.728 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.42027.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.42027.log.gz 2024-09-22T03:24:17.729 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.30646.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30646.log.gz 2024-09-22T03:24:17.790 INFO:teuthology.orchestra.run.smithi153.stderr: 94.1% -- replaced with /var/log/ceph/ceph-client.0.43901.log.gz 2024-09-22T03:24:17.850 INFO:teuthology.orchestra.run.smithi153.stderr: 94.1% -- replaced with /var/log/ceph/ceph-client.0.33956.log.gz 2024-09-22T03:24:17.968 INFO:teuthology.orchestra.run.smithi153.stderr: 94.1% -- replaced with /var/log/ceph/ceph-client.0.41696.log.gz 2024-09-22T03:24:18.033 INFO:teuthology.orchestra.run.smithi153.stderr: 94.1% -- replaced with /var/log/ceph/ceph-client.0.37783.log.gz 2024-09-22T03:24:18.050 INFO:teuthology.orchestra.run.smithi153.stderr: 89.9% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-09-22T03:24:34.019 INFO:teuthology.orchestra.run.smithi196.stderr: 92.8% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-09-22T03:24:35.424 INFO:teuthology.orchestra.run.smithi153.stderr: 92.8% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-09-22T03:24:35.593 INFO:teuthology.orchestra.run.smithi196.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-09-22T03:24:35.595 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-09-22T03:24:35.595 INFO:teuthology.orchestra.run.smithi196.stderr:real 0m18.414s 2024-09-22T03:24:35.595 INFO:teuthology.orchestra.run.smithi196.stderr:user 0m34.041s 2024-09-22T03:24:35.595 INFO:teuthology.orchestra.run.smithi196.stderr:sys 0m1.240s 2024-09-22T03:24:35.969 INFO:teuthology.orchestra.run.smithi153.stderr: 92.8% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-09-22T03:24:35.971 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-22T03:24:35.971 INFO:teuthology.orchestra.run.smithi153.stderr:real 0m18.833s 2024-09-22T03:24:35.971 INFO:teuthology.orchestra.run.smithi153.stderr:user 0m39.698s 2024-09-22T03:24:35.971 INFO:teuthology.orchestra.run.smithi153.stderr:sys 0m1.666s 2024-09-22T03:24:35.971 INFO:tasks.ceph:Archiving logs... 2024-09-22T03:24:35.971 DEBUG:teuthology.misc:Transferring archived files from smithi153:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-09-21_21:56:02-crimson-rados-squid-distro-crimson-smithi/7914725/remote/smithi153/log 2024-09-22T03:24:35.972 DEBUG:teuthology.orchestra.run.smithi153:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-22T03:24:42.084 DEBUG:teuthology.misc:Transferring archived files from smithi196:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-09-21_21:56:02-crimson-rados-squid-distro-crimson-smithi/7914725/remote/smithi196/log 2024-09-22T03:24:42.085 DEBUG:teuthology.orchestra.run.smithi196:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-22T03:24:47.307 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-09-22T03:24:47.314 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-09-22T03:24:47.314 DEBUG:teuthology.orchestra.run.smithi153:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-22T03:24:47.317 DEBUG:teuthology.orchestra.run.smithi196:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-22T03:24:47.410 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-22T03:24:47.423 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-22T03:24:47.423 DEBUG:teuthology.orchestra.run.smithi153:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-22T03:24:47.426 DEBUG:teuthology.orchestra.run.smithi196:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-22T03:24:47.441 INFO:teuthology.orchestra.run.smithi153.stderr:bash: line 1: ntpq: command not found 2024-09-22T03:24:47.442 INFO:teuthology.orchestra.run.smithi196.stderr:bash: line 1: ntpq: command not found 2024-09-22T03:24:47.446 INFO:teuthology.orchestra.run.smithi153.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-22T03:24:47.446 INFO:teuthology.orchestra.run.smithi153.stdout:=============================================================================== 2024-09-22T03:24:47.446 INFO:teuthology.orchestra.run.smithi153.stdout:^* hv01.front.sepia.ceph.com 3 6 377 65 -156us[ -172us] +/- 43ms 2024-09-22T03:24:47.446 INFO:teuthology.orchestra.run.smithi153.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 65 +414us[ +414us] +/- 67ms 2024-09-22T03:24:47.446 INFO:teuthology.orchestra.run.smithi153.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 64 -302us[ -302us] +/- 68ms 2024-09-22T03:24:47.446 INFO:teuthology.orchestra.run.smithi153.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-09-22T03:24:47.447 INFO:teuthology.orchestra.run.smithi196.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-22T03:24:47.447 INFO:teuthology.orchestra.run.smithi196.stdout:=============================================================================== 2024-09-22T03:24:47.447 INFO:teuthology.orchestra.run.smithi196.stdout:^* hv01.front.sepia.ceph.com 3 7 377 126 -136us[ -141us] +/- 42ms 2024-09-22T03:24:47.448 INFO:teuthology.orchestra.run.smithi196.stdout:^+ hv02.front.sepia.ceph.com 3 7 377 66 +455us[ +455us] +/- 67ms 2024-09-22T03:24:47.448 INFO:teuthology.orchestra.run.smithi196.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 63 -282us[ -282us] +/- 68ms 2024-09-22T03:24:47.448 INFO:teuthology.orchestra.run.smithi196.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-09-22T03:24:47.448 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-22T03:24:47.456 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-22T03:24:47.457 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-22T03:24:47.467 DEBUG:teuthology.orchestra.run.smithi153:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-22T03:24:47.564 DEBUG:teuthology.orchestra.run.smithi196:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-22T03:24:47.631 DEBUG:teuthology.orchestra.run.smithi153:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-22T03:24:47.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-22T03:24:47.660 DEBUG:teuthology.orchestra.run.smithi196:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-22T03:24:47.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-22T03:24:47.689 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-22T03:24:47.701 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-22T03:24:47.735 INFO:teuthology.task.internal:Duration was 1467.678792 seconds 2024-09-22T03:24:47.736 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-22T03:24:47.753 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-22T03:24:47.753 DEBUG:teuthology.orchestra.run.smithi153:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-22T03:24:47.756 DEBUG:teuthology.orchestra.run.smithi196:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-22T03:24:47.793 INFO:teuthology.orchestra.run.smithi153.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-22T03:24:47.796 INFO:teuthology.orchestra.run.smithi196.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-22T03:24:48.110 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-22T03:24:48.111 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi153.front.sepia.ceph.com 2024-09-22T03:24:48.111 DEBUG:teuthology.orchestra.run.smithi153:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-22T03:24:48.171 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi196.front.sepia.ceph.com 2024-09-22T03:24:48.171 DEBUG:teuthology.orchestra.run.smithi196:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-22T03:24:48.195 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-22T03:24:48.195 DEBUG:teuthology.orchestra.run.smithi153:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-22T03:24:48.214 DEBUG:teuthology.orchestra.run.smithi196:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-22T03:24:48.261 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-22T03:24:48.261 DEBUG:teuthology.orchestra.run.smithi153:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-22T03:24:48.281 DEBUG:teuthology.orchestra.run.smithi196:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-22T03:24:48.405 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-22T03:24:48.414 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-22T03:24:48.414 DEBUG:teuthology.orchestra.run.smithi153:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-22T03:24:48.439 DEBUG:teuthology.orchestra.run.smithi196:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-22T03:24:48.463 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-22T03:24:48.473 DEBUG:teuthology.orchestra.run.smithi153:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-22T03:24:48.482 DEBUG:teuthology.orchestra.run.smithi196:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-22T03:24:48.502 INFO:teuthology.orchestra.run.smithi153.stdout:kernel.core_pattern = core 2024-09-22T03:24:48.526 INFO:teuthology.orchestra.run.smithi196.stdout:kernel.core_pattern = core 2024-09-22T03:24:48.579 DEBUG:teuthology.orchestra.run.smithi153:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-22T03:24:48.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-22T03:24:48.593 DEBUG:teuthology.orchestra.run.smithi196:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-22T03:24:48.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-22T03:24:48.633 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-22T03:24:48.644 INFO:teuthology.task.internal:Transferring archived files... 2024-09-22T03:24:48.644 DEBUG:teuthology.misc:Transferring archived files from smithi153:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-09-21_21:56:02-crimson-rados-squid-distro-crimson-smithi/7914725/remote/smithi153 2024-09-22T03:24:48.645 DEBUG:teuthology.orchestra.run.smithi153:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-22T03:24:48.678 DEBUG:teuthology.misc:Transferring archived files from smithi196:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-09-21_21:56:02-crimson-rados-squid-distro-crimson-smithi/7914725/remote/smithi196 2024-09-22T03:24:48.678 DEBUG:teuthology.orchestra.run.smithi196:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-22T03:24:48.711 INFO:teuthology.task.internal:Removing archive directory... 2024-09-22T03:24:48.711 DEBUG:teuthology.orchestra.run.smithi153:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-22T03:24:48.714 DEBUG:teuthology.orchestra.run.smithi196:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-22T03:24:48.761 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-22T03:24:48.771 INFO:teuthology.task.internal:Not uploading archives. 2024-09-22T03:24:48.772 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-22T03:24:48.785 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-22T03:24:48.785 DEBUG:teuthology.orchestra.run.smithi153:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-22T03:24:48.788 DEBUG:teuthology.orchestra.run.smithi196:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-22T03:24:48.800 INFO:teuthology.orchestra.run.smithi153.stdout: 265329 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 22 03:24 /home/ubuntu/cephtest 2024-09-22T03:24:48.816 INFO:teuthology.orchestra.run.smithi196.stdout: 265324 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 22 03:24 /home/ubuntu/cephtest 2024-09-22T03:24:48.817 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-22T03:24:48.828 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-22T03:24:48.860 INFO:teuthology.run:Summary data: description: crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/radosbench-high-concurrency} duration: 1467.6787917613983 flavor: crimson owner: scheduled_teuthology@teuthology success: true 2024-09-22T03:24:48.861 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-22T03:24:48.937 INFO:teuthology.run:pass